Sep 04 22:08:36.376382 ip-10-0-139-78 systemd[1]: Starting Kubernetes Kubelet...
Sep 04 22:08:36.727649 ip-10-0-139-78 kubenswrapper[2576]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 22:08:36.727649 ip-10-0-139-78 kubenswrapper[2576]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 04 22:08:36.727649 ip-10-0-139-78 kubenswrapper[2576]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 22:08:36.727649 ip-10-0-139-78 kubenswrapper[2576]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 04 22:08:36.727649 ip-10-0-139-78 kubenswrapper[2576]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 04 22:08:36.729145 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.729066    2576 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 04 22:08:36.736500 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736477    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:08:36.736500 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736497    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:08:36.736500 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736501    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:08:36.736500 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736504    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:08:36.736500 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736507    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736511    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736514    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736517    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736520    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736523    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736525    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736528    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736532    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736535    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736537    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736540    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736543    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736545    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736548    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736551    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736553    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736556    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736559    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736561    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:08:36.736705 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736564    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736566    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736569    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736571    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736574    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736576    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736579    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736583    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736585    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736588    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736590    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736593    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736595    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736598    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736601    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736604    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736606    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736609    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736614    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:08:36.737208 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736619    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736622    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736624    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736627    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736629    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736632    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736634    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736637    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736639    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736642    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736645    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736647    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736650    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736652    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736655    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736657    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736659    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736662    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736664    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736668    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:08:36.737681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736671    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736673    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736676    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736678    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736681    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736684    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736687    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736690    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736693    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736696    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736701    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736704    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736707    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736710    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736714    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736717    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736720    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736723    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736725    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736728    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:08:36.738176 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736731    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736734    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736737    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736740    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736742    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736745    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736748    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736750    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736754    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736759    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736762    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736765    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736768    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736771    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736773    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.736776    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737494    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737502    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737505    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:08:36.738660 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737508    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737510    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737513    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737516    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737518    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737521    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737523    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737526    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737529    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737532    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737534    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737537    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737539    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737541    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737545    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737547    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737550    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737552    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737555    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737557    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:08:36.739236 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737560    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737563    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737565    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737568    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737570    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737573    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737576    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737578    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737581    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737584    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737586    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737589    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737592    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737594    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737597    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737599    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737602    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737605    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737608    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737610    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:08:36.739759 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737613    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737616    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737619    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737621    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737626    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737630    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737633    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737636    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737639    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737642    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737645    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737649    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737653    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737656    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737658    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737662    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737666    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737669    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737672    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:08:36.740285 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737675    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737678    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737681    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737684    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737687    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737690    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737693    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737697    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737700    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737702    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737705    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737707    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737710    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737712    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737715    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737718    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737720    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737723    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737725    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737727    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:08:36.740754 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737731    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737736    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737738    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737741    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737743    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737747    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737750    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737752    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737754    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737757    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737760    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737762    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737765    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737768    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737771    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737774    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.737776    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738158    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738163    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738166    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:08:36.741308 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738169    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738172    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738175    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738178    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738181    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738183    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738186    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738189    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738192    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738195    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738197    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738200    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738202    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738206    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738208    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738211    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738213    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738216    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738218    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738221    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:08:36.741805 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738223    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738226    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738229    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738231    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738234    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738237    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738240    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738243    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738245    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738248    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738251    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738254    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738256    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738259    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738261    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738265    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738268    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738271    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738274    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738276    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:08:36.742313 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738279    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738282    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738285    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738287    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738290    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738292    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738296    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738298    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738301    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738303    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738306    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738308    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738311    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738313    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738316    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738319    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738322    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738324    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738329    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738332    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:08:36.742850 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738334    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738337    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738339    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738342    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738345    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738347    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738350    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738353    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738355    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738358    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738360    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738363    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738365    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738367    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738370    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738374    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738378    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738381    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738383    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:08:36.743350 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738387    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738390    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738393    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738404    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738408    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738411    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738414    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738419    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738422    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738425    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738427    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738431    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738434    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738437    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738439    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738442    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.738444    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738518    2576 flags.go:64] FLAG: --address="0.0.0.0"
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738552    2576 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738558    2576 flags.go:64] FLAG: --anonymous-auth="true"
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738563    2576 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 04 22:08:36.743832 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738567    2576 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738570    2576 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738574    2576 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738578    2576 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738581    2576 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738584    2576 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738588    2576 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738591    2576 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738594    2576 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738597    2576 flags.go:64] FLAG: --cgroup-root=""
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738600    2576 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738603    2576 flags.go:64] FLAG: --client-ca-file=""
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738610    2576 flags.go:64] FLAG: --cloud-provider="external"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738613    2576 flags.go:64] FLAG: --cluster-dns="[]"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738619    2576 flags.go:64] FLAG: --cluster-domain=""
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738622    2576 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738625    2576 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738628    2576 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738633    2576 flags.go:64] FLAG: --container-log-max-files="5"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738637    2576 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738640    2576 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738643    2576 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738647    2576 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738650    2576 flags.go:64] FLAG: --contention-profiling="false"
Sep 04 22:08:36.744384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738653    2576 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738656    2576 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738661    2576 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738664    2576 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738668    2576 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738672    2576 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738674    2576 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738677    2576 flags.go:64] FLAG: --enable-load-reader="false"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738680    2576 flags.go:64] FLAG: --enable-server="true"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738683    2576 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738688    2576 flags.go:64] FLAG: --event-burst="100"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738692    2576 flags.go:64] FLAG: --event-qps="50"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738694    2576 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738697    2576 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738700    2576 flags.go:64] FLAG: --eviction-hard=""
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738704    2576 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738707    2576 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738710    2576 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738713    2576 flags.go:64] FLAG: --eviction-soft=""
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738715    2576 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738718    2576 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738722    2576 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738725    2576 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738727    2576 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738730    2576 flags.go:64] FLAG: --fail-swap-on="true"
Sep 04 22:08:36.744964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738733    2576 flags.go:64] FLAG: --feature-gates=""
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738739    2576 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738742    2576 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738745    2576 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738748    2576 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738753    2576 flags.go:64] FLAG: --healthz-port="10248"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738756    2576 flags.go:64] FLAG: --help="false"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738759    2576 flags.go:64] FLAG: --hostname-override="ip-10-0-139-78.ec2.internal"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738762    2576 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738765    2576 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738768    2576 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738771    2576 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738775    2576 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738777    2576 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738780    2576 flags.go:64] FLAG: --image-service-endpoint=""
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738783    2576 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738786    2576 flags.go:64] FLAG: --kube-api-burst="100"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738789    2576 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738792    2576 flags.go:64] FLAG: --kube-api-qps="50"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738795    2576 flags.go:64] FLAG: --kube-reserved=""
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738798    2576 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738800    2576 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738803    2576 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738806    2576 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 04 22:08:36.745624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738809    2576 flags.go:64] FLAG: --lock-file=""
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738812    2576 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738815    2576 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738818    2576 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738823    2576 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738826    2576 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738829    2576 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738831    2576 flags.go:64] FLAG: --logging-format="text"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738834    2576 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738839    2576 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738842    2576 flags.go:64] FLAG: --manifest-url=""
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738845    2576 flags.go:64] FLAG: --manifest-url-header=""
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738849    2576 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738853    2576 flags.go:64] FLAG: --max-open-files="1000000"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738857    2576 flags.go:64] FLAG: --max-pods="110"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738859    2576 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738862    2576 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738865    2576 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738868    2576 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738872    2576 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738874    2576 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738877    2576 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738885    2576 flags.go:64] FLAG: --node-status-max-images="50"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738890    2576 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738894    2576 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 04 22:08:36.746382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738897    2576 flags.go:64] FLAG: --pod-cidr=""
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738899    2576 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738905    2576 flags.go:64] FLAG: --pod-manifest-path=""
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738907    2576 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738910    2576 flags.go:64] FLAG: --pods-per-core="0"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738913    2576 flags.go:64] FLAG: --port="10250"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738916    2576 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738919    2576 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00f4b7ce83a31fa46"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738922    2576 flags.go:64] FLAG: --qos-reserved=""
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738925    2576 flags.go:64] FLAG: --read-only-port="10255"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738928    2576 flags.go:64] FLAG: --register-node="true"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738931    2576 flags.go:64] FLAG: --register-with-taints=""
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738934    2576 flags.go:64] FLAG: --registry-burst="10"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738940    2576 flags.go:64] FLAG: --registry-qps="5"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738943    2576 flags.go:64] FLAG: --reserved-cpus=""
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738946    2576 flags.go:64] FLAG: --reserved-memory=""
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738951    2576 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738954    2576 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738957    2576 flags.go:64] FLAG: --rotate-certificates="false"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738960    2576 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738962    2576 flags.go:64] FLAG: --runonce="false"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738966    2576 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738969    2576 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738972    2576 flags.go:64] FLAG: --seccomp-default="false"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738975    2576 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 04 22:08:36.747475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738978    2576 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738981    2576 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738984    2576 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738988    2576 flags.go:64] FLAG: --storage-driver-password="root"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738990    2576 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738994    2576 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738996    2576 flags.go:64] FLAG: --storage-driver-user="root"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.738999    2576 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739002    2576 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739005    2576 flags.go:64] FLAG: --system-cgroups=""
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739008    2576 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739013    2576 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739016    2576 flags.go:64] FLAG: --tls-cert-file=""
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739018    2576 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739022    2576 flags.go:64] FLAG: --tls-min-version=""
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739025    2576 flags.go:64] FLAG: --tls-private-key-file=""
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739028    2576 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739030    2576 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739033    2576 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739036    2576 flags.go:64] FLAG: --v="2"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739040    2576 flags.go:64] FLAG: --version="false"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739047    2576 flags.go:64] FLAG: --vmodule=""
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739063    2576 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.739068    2576 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739176    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:08:36.748585 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739180    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739183    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739186    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739190    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739192    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739195    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739198    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739200    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739203    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739206    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739208    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739211    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739213    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739216    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739219    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739221    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739224    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739226    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739231    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:08:36.749797 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739234    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739237    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739240    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739243    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739246    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739249    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739252    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739254    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739257    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739260    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739268    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739272    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739275    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739277    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739280    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739282    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739286    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739288    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739291    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739294    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:08:36.750312 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739298    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739301    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739304    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739307    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739310    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739313    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739315    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739318    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739320    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739323    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739325    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739328    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739331    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739333    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739336    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739338    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739341    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739343    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739345    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739348    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:08:36.751107 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739351    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739353    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739356    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739363    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739366    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739368    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739371    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739373    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739377    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739380    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739383    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739385    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739388    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739390    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739394    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739397    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739400    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739403    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739405    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:08:36.751786 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739408    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739410    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739413    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739415    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739418    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739420    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739423    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739425    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739427    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739430    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739432    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739435    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739437    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739440    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739443    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739445    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739449    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739454    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739457    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:08:36.752314 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.739459    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.740158    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.748294    2576 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.748316    2576 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.748343    2576 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.748352    2576 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748437    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748445    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748450    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748455    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748460    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748466    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748470    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748474    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:08:36.752961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748478    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748483    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748487    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748495    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748500    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748506    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748510    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748515    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748519    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748524    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748529    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748533    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748537    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748542    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748546    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748550    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748554    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748559    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748563    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748567    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:08:36.753564 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748571    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748577    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748581    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748585    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748591    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748597    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748601    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748606    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748610    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748614    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748618    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748623    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748628    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748632    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748636    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748640    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748644    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748648    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748652    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748657    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:08:36.754069 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748661    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748664    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748669    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748673    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748677    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748681    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748685    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748689    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748693    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748698    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748702    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748706    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748710    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748715    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748723    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748728    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748732    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748736    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748740    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:08:36.754620 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748745    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748749    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748753    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748757    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748761    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748765    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748769    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748773    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748777    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748781    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748786    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748791    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748795    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748799    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748803    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748808    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748812    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748816    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748820    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748824    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:08:36.755184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748829    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748833    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748837    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748842    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748846    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748850    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748854    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748859    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748863    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748868    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748873    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.748876    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.748885    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749081    2576 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749090    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 04 22:08:36.755721 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749094    2576 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749099    2576 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749104    2576 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749108    2576 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749112    2576 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749117    2576 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749121    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749124    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749129    2576 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749132    2576 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749136    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749139    2576 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749143    2576 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749147    2576 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749151    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749155    2576 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749159    2576 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749162    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749166    2576 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749170    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 04 22:08:36.756125 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749174    2576 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749177    2576 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749181    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749185    2576 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749190    2576 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749194    2576 feature_gate.go:328] unrecognized feature gate: Example2
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749199    2576 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749204    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749208    2576 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749212    2576 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749218    2576 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749223    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749228    2576 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749232    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749237    2576 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749241    2576 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749246    2576 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749250    2576 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749255    2576 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749259    2576 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 04 22:08:36.756819 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749263    2576 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749267    2576 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749271    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749276    2576 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749280    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749284    2576 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749288    2576 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749295    2576 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749307    2576 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749312    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749317    2576 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749322    2576 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749326    2576 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749330    2576 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749335    2576 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749339    2576 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749343    2576 feature_gate.go:328] unrecognized feature gate: Example
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749348    2576 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749352    2576 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749357    2576 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 04 22:08:36.757687 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749361    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749365    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749370    2576 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749373    2576 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749378    2576 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749382    2576 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749386    2576 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749390    2576 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749394    2576 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749398    2576 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749403    2576 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749407    2576 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749411    2576 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749415    2576 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749420    2576 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749424    2576 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749429    2576 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749433    2576 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749437    2576 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749441    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 04 22:08:36.758560 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749446    2576 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749450    2576 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749455    2576 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749459    2576 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749463    2576 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749470    2576 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749475    2576 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749479    2576 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749483    2576 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749487    2576 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749493    2576 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749497    2576 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749503    2576 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749507    2576 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749511    2576 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749515    2576 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 04 22:08:36.759204 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:36.749520    2576 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 04 22:08:36.759651 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.749528    2576 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 04 22:08:36.759651 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.749692    2576 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 04 22:08:36.759651 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.753250    2576 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 04 22:08:36.759651 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.754178    2576 server.go:1019] "Starting client certificate rotation"
Sep 04 22:08:36.759651 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.754273    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 04 22:08:36.759651 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.754312    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 04 22:08:36.759651 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.756996    2576 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 04 22:08:36.759651 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.758765    2576 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 04 22:08:36.770499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.770480    2576 log.go:25] "Validated CRI v1 runtime API"
Sep 04 22:08:36.773436 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.773422    2576 log.go:25] "Validated CRI v1 image API"
Sep 04 22:08:36.774234 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.774214    2576 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 04 22:08:36.776849 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.776822    2576 fs.go:135] Filesystem UUIDs: map[70dcd009-c301-48b9-9c9b-8257a1425a65:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 7a7467f4-ee15-4238-a4f8-be4a37eb9128:/dev/nvme0n1p4]
Sep 04 22:08:36.776924 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.776847    2576 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 04 22:08:36.782321 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.782207    2576 manager.go:216] Machine: {Timestamp:2026-09-04 22:08:36.780985589 +0000 UTC m=+0.299708115 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096680 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25be5ad0004b7a142390000083da4b SystemUUID:ec25be5a-d000-4b7a-1423-90000083da4b BootID:d1e79c9f-d267-4e46-983e-a53088fc5bc5 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:af:87:f9:03:f3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:af:87:f9:03:f3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ca:f2:f6:8f:d3:4c Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 04 22:08:36.782321 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.782316    2576 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 04 22:08:36.782427 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.782398    2576 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 04 22:08:36.785149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.785124    2576 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 04 22:08:36.785298 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.785150    2576 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-78.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 04 22:08:36.786483 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.786472    2576 topology_manager.go:138] "Creating topology manager with none policy"
Sep 04 22:08:36.786522 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.786488    2576 container_manager_linux.go:309] "Creating device plugin manager"
Sep 04 22:08:36.786522 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.786496    2576 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 04 22:08:36.787126 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.787116    2576 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 04 22:08:36.788203 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.788194    2576 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 04 22:08:36.789679 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.789663    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 04 22:08:36.792400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.792388    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 04 22:08:36.792513 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.792504    2576 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 04 22:08:36.792546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.792534    2576 kubelet.go:486] "Attempting to sync node with API server"
Sep 04 22:08:36.792546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.792544    2576 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 04 22:08:36.792601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.792555    2576 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 04 22:08:36.792601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.792564    2576 kubelet.go:398] "Adding apiserver pod source"
Sep 04 22:08:36.792601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.792574    2576 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 04 22:08:36.792966 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.792953    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 04 22:08:36.792999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.792979    2576 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 04 22:08:36.795202 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.795185    2576 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 04 22:08:36.797702 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.797685    2576 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 04 22:08:36.797789 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.797723    2576 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 04 22:08:36.799392 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799376    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 04 22:08:36.799480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799398    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 04 22:08:36.799480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799409    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 04 22:08:36.799480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799417    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 04 22:08:36.799480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799428    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 04 22:08:36.799480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799436    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 04 22:08:36.799480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799455    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 04 22:08:36.799480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799463    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 04 22:08:36.799480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799473    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 04 22:08:36.799480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799481    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 04 22:08:36.799937 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799505    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 04 22:08:36.799988 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.799942    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 04 22:08:36.800711 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.800701    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 04 22:08:36.800760 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.800713    2576 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 04 22:08:36.804257 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.804242    2576 server.go:1268] "Started kubelet"
Sep 04 22:08:36.804339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.804313    2576 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 04 22:08:36.804382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.804315    2576 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 04 22:08:36.804420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.804380    2576 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 04 22:08:36.805008 ip-10-0-139-78 systemd[1]: Started Kubernetes Kubelet.
Sep 04 22:08:36.805578 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.805561    2576 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 04 22:08:36.810713 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.810693    2576 server.go:310] "Adding debug handlers to kubelet server"
Sep 04 22:08:36.812017 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.811990    2576 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 04 22:08:36.814000 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.813981    2576 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 04 22:08:36.814592 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.814564    2576 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 04 22:08:36.814696 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.814646    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:36.815159 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.815141    2576 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 04 22:08:36.815159 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.815143    2576 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 04 22:08:36.815288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.815170    2576 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 04 22:08:36.815288 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.815269    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 04 22:08:36.815398 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.815296    2576 reconstruct.go:97] "Volume reconstruction finished"
Sep 04 22:08:36.815398 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.815312    2576 reconciler.go:29] "Reconciler: start to sync state"
Sep 04 22:08:36.815398 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.815381    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 04 22:08:36.815853 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.815837    2576 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-78.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 04 22:08:36.816267 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.815372    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-78.ec2.internal.18d23d9613b041cb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-78.ec2.internal,UID:ip-10-0-139-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-78.ec2.internal,},FirstTimestamp:2026-09-04 22:08:36.804215243 +0000 UTC m=+0.322937767,LastTimestamp:2026-09-04 22:08:36.804215243 +0000 UTC m=+0.322937767,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-78.ec2.internal,}"
Sep 04 22:08:36.816588 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.816574    2576 factory.go:55] Registering systemd factory
Sep 04 22:08:36.816659 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.816591    2576 factory.go:223] Registration of the systemd container factory successfully
Sep 04 22:08:36.816824 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.816805    2576 factory.go:153] Registering CRI-O factory
Sep 04 22:08:36.816824 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.816823    2576 factory.go:223] Registration of the crio container factory successfully
Sep 04 22:08:36.816979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.816863    2576 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 04 22:08:36.816979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.816877    2576 factory.go:103] Registering Raw factory
Sep 04 22:08:36.816979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.816888    2576 manager.go:1196] Started watching for new ooms in manager
Sep 04 22:08:36.817327 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.817254    2576 manager.go:349] Starting recovery of all containers
Sep 04 22:08:36.827415 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.827244    2576 manager.go:354] Recovery completed
Sep 04 22:08:36.828703 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.828679    2576 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-78.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 04 22:08:36.828923 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.828827    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 04 22:08:36.832435 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.832421    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:08:36.834740 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.834712    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:08:36.834819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.834754    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:08:36.834819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.834767    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:08:36.835386 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.835366    2576 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 04 22:08:36.835386 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.835380    2576 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 04 22:08:36.835386 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.835387    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:08:36.835548 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.835398    2576 state_mem.go:36] "Initialized new in-memory state store"
Sep 04 22:08:36.836143 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.836127    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:08:36.836205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.836159    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:08:36.836205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.836177    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:08:36.837436 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.837423    2576 policy_none.go:49] "None policy: Start"
Sep 04 22:08:36.837495 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.837441    2576 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 04 22:08:36.837495 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.837454    2576 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 04 22:08:36.838613 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.838529    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-78.ec2.internal.18d23d961582024c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-78.ec2.internal,UID:ip-10-0-139-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-78.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-78.ec2.internal,},FirstTimestamp:2026-09-04 22:08:36.834738764 +0000 UTC m=+0.353461288,LastTimestamp:2026-09-04 22:08:36.834738764 +0000 UTC m=+0.353461288,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-78.ec2.internal,}"
Sep 04 22:08:36.839364 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.839350    2576 policy_none.go:47] "Start"
Sep 04 22:08:36.852100 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.852016    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-78.ec2.internal.18d23d96158259df  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-78.ec2.internal,UID:ip-10-0-139-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-78.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-78.ec2.internal,},FirstTimestamp:2026-09-04 22:08:36.834761183 +0000 UTC m=+0.353483710,LastTimestamp:2026-09-04 22:08:36.834761183 +0000 UTC m=+0.353483710,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-78.ec2.internal,}"
Sep 04 22:08:36.865454 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.865379    2576 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-78.ec2.internal.18d23d9615828266  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-78.ec2.internal,UID:ip-10-0-139-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-78.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-78.ec2.internal,},FirstTimestamp:2026-09-04 22:08:36.834771558 +0000 UTC m=+0.353494082,LastTimestamp:2026-09-04 22:08:36.834771558 +0000 UTC m=+0.353494082,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-78.ec2.internal,}"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.866773    2576 manager.go:337] "Starting Device Plugin manager"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.866810    2576 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.866834    2576 server.go:87] "Starting device plugin registration server"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.867189    2576 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.867202    2576 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.867309    2576 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.867397    2576 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.867406    2576 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.867999    2576 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.868099    2576 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:36.891358 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.879016    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-78.ec2.internal.18d23d961582024c\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-78.ec2.internal.18d23d961582024c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-78.ec2.internal,UID:ip-10-0-139-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-78.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-78.ec2.internal,},FirstTimestamp:2026-09-04 22:08:36.834738764 +0000 UTC m=+0.353461288,LastTimestamp:2026-09-04 22:08:36.836143496 +0000 UTC m=+0.354866021,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-78.ec2.internal,}"
Sep 04 22:08:36.895943 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.895869    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-78.ec2.internal.18d23d96158259df\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-78.ec2.internal.18d23d96158259df  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-78.ec2.internal,UID:ip-10-0-139-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-139-78.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-139-78.ec2.internal,},FirstTimestamp:2026-09-04 22:08:36.834761183 +0000 UTC m=+0.353483710,LastTimestamp:2026-09-04 22:08:36.836169375 +0000 UTC m=+0.354891901,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-78.ec2.internal,}"
Sep 04 22:08:36.896737 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.896722    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gvkp5"
Sep 04 22:08:36.907610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.907589    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gvkp5"
Sep 04 22:08:36.908355 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.908277    2576 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-139-78.ec2.internal.18d23d9615828266\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-78.ec2.internal.18d23d9615828266  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-78.ec2.internal,UID:ip-10-0-139-78.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-78.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-78.ec2.internal,},FirstTimestamp:2026-09-04 22:08:36.834771558 +0000 UTC m=+0.353494082,LastTimestamp:2026-09-04 22:08:36.836182274 +0000 UTC m=+0.354904802,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-78.ec2.internal,}"
Sep 04 22:08:36.910352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.910318    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 04 22:08:36.911487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.911471    2576 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 04 22:08:36.911564 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.911515    2576 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 04 22:08:36.911564 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.911537    2576 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 04 22:08:36.911695 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.911622    2576 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 04 22:08:36.921639 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.921621    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:08:36.967476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.967449    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:08:36.968427 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.968408    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:08:36.968535 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.968443    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:08:36.968535 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.968457    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:08:36.968535 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.968484    2576 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-78.ec2.internal"
Sep 04 22:08:36.981389 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:36.981305    2576 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-78.ec2.internal"
Sep 04 22:08:36.981389 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:36.981332    2576 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-78.ec2.internal\": node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.012186 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.012157    2576 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal"]
Sep 04 22:08:37.012288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.012240    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:08:37.014171 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.014154    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:08:37.014237 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.014187    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:08:37.014237 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.014200    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:08:37.016383 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.016366    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ea6adedb9dcd115be4eba74ebae07fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal\" (UID: \"3ea6adedb9dcd115be4eba74ebae07fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.016434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.016392    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea6adedb9dcd115be4eba74ebae07fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal\" (UID: \"3ea6adedb9dcd115be4eba74ebae07fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.016434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.016403    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:08:37.016510 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.016500    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.016543 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.016524    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:08:37.017540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.017522    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:08:37.017635 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.017550    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:08:37.017635 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.017522    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:08:37.017635 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.017561    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:08:37.017635 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.017598    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:08:37.017635 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.017618    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:08:37.019716 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.019702    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.019775 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.019726    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:08:37.020444 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.020426    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:08:37.020531 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.020460    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:08:37.020531 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.020475    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:08:37.044861 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.044831    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-78.ec2.internal\" not found" node="ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.046226 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.046207    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.049254 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.049240    2576 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-78.ec2.internal\" not found" node="ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.117141 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.117114    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ea6adedb9dcd115be4eba74ebae07fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal\" (UID: \"3ea6adedb9dcd115be4eba74ebae07fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.117141 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.117143    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea6adedb9dcd115be4eba74ebae07fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal\" (UID: \"3ea6adedb9dcd115be4eba74ebae07fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.117272 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.117162    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/354628df5e5c0356cd158ffc3ea8a19c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-78.ec2.internal\" (UID: \"354628df5e5c0356cd158ffc3ea8a19c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.117272 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.117224    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ea6adedb9dcd115be4eba74ebae07fc-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal\" (UID: \"3ea6adedb9dcd115be4eba74ebae07fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.117272 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.117250    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3ea6adedb9dcd115be4eba74ebae07fc-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal\" (UID: \"3ea6adedb9dcd115be4eba74ebae07fc\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.146791 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.146766    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.217471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.217440    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/354628df5e5c0356cd158ffc3ea8a19c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-78.ec2.internal\" (UID: \"354628df5e5c0356cd158ffc3ea8a19c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.217551 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.217504    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/354628df5e5c0356cd158ffc3ea8a19c-config\") pod \"kube-apiserver-proxy-ip-10-0-139-78.ec2.internal\" (UID: \"354628df5e5c0356cd158ffc3ea8a19c\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.247535 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.247481    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.347350 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.347318    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.348403 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.348386    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.353656 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.353628    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:37.449098 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.449045    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.551263 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.551227    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.653715 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.653680    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.753964 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.753925    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.753964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.753959    2576 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 04 22:08:37.754505 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.754116    2576 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 04 22:08:37.854719 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.854688    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.912958 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.912904    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-03 22:03:37 +0000 UTC" deadline="2028-05-19 03:17:15.060692518 +0000 UTC"
Sep 04 22:08:37.912958 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.912955    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14933h8m37.147743764s"
Sep 04 22:08:37.936474 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.936441    2576 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 04 22:08:37.937473 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.937453    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientMemory"
Sep 04 22:08:37.937593 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.937487    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasNoDiskPressure"
Sep 04 22:08:37.937593 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.937498    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeHasSufficientPID"
Sep 04 22:08:37.955943 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:37.955896    2576 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-78.ec2.internal\" not found"
Sep 04 22:08:37.967296 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:37.967260    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ea6adedb9dcd115be4eba74ebae07fc.slice/crio-dd0edb6c6d3c0eb4e6e766c128979036514d8190a93e5c50cf5010ab4fd78c0b WatchSource:0}: Error finding container dd0edb6c6d3c0eb4e6e766c128979036514d8190a93e5c50cf5010ab4fd78c0b: Status 404 returned error can't find the container with id dd0edb6c6d3c0eb4e6e766c128979036514d8190a93e5c50cf5010ab4fd78c0b
Sep 04 22:08:37.967518 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:37.967499    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod354628df5e5c0356cd158ffc3ea8a19c.slice/crio-df517c476e2dc237e6a3e2862c5b365e50014111ac8f8795b1dda593424aa32e WatchSource:0}: Error finding container df517c476e2dc237e6a3e2862c5b365e50014111ac8f8795b1dda593424aa32e: Status 404 returned error can't find the container with id df517c476e2dc237e6a3e2862c5b365e50014111ac8f8795b1dda593424aa32e
Sep 04 22:08:37.972852 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:37.972836    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 22:08:38.053223 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.053195    2576 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:08:38.054504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.054490    2576 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:08:38.079888 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.079810    2576 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:08:38.116475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.116426    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:38.140026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.140000    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 04 22:08:38.140949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.140935    2576 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal"
Sep 04 22:08:38.153613 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.153597    2576 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 04 22:08:38.793869 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.793838    2576 apiserver.go:52] "Watching apiserver"
Sep 04 22:08:38.805402 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.805370    2576 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 04 22:08:38.806517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.806490    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-ftrbj","openshift-multus/multus-additional-cni-plugins-slx5l","openshift-network-operator/iptables-alerter-6587w","openshift-ovn-kubernetes/ovnkube-node-wrqjr","kube-system/global-pull-secret-syncer-8hpxw","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn","openshift-cluster-node-tuning-operator/tuned-7v6rc","openshift-dns/node-resolver-5mt7k","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal","openshift-multus/multus-pns4q","openshift-multus/network-metrics-daemon-ssvlz","openshift-network-diagnostics/network-check-target-kr6k4","kube-system/konnectivity-agent-5d6dc","kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal"]
Sep 04 22:08:38.809404 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.809269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.811628 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.811575    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:38.811754 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:38.811673    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:38.813170 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.813144    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 04 22:08:38.813267 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.813239    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-klqb5\""
Sep 04 22:08:38.813855 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.813838    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.814995 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.814975    2576 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 04 22:08:38.815287 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.815264    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 04 22:08:38.816858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.816326    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.820107 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.820086    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 04 22:08:38.820262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.820229    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 04 22:08:38.820411 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.820350    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 04 22:08:38.820500 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.820463    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 04 22:08:38.820820 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.820800    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-8gqfs\""
Sep 04 22:08:38.821038 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.821024    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:38.822278 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.822259    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 04 22:08:38.822922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.822736    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-m4z7k\""
Sep 04 22:08:38.823354 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.823309    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:38.825103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.824875    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lnwtj\""
Sep 04 22:08:38.825103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.824878    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 04 22:08:38.825103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.824973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-etc-openvswitch\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825007    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-log-socket\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-ovnkube-config\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825059    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-sysconfig\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.825103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825082    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-kubernetes\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.824884    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825148    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825175    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c0c7485-f8c7-4a05-a496-9e8604e77b66-host-slash\") pod \"iptables-alerter-6587w\" (UID: \"6c0c7485-f8c7-4a05-a496-9e8604e77b66\") " pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-slash\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-var-lib-openvswitch\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825262    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-run-ovn\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825286    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-node-log\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnjfn\" (UniqueName: \"kubernetes.io/projected/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-kube-api-access-xnjfn\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825393    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-cni-bin\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825425    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-cni-netd\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825455    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-env-overrides\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.825487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-sysctl-conf\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825571    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-run\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825609    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-sys\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825652    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-tmp\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825689    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825715    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-lib-modules\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fthrb\" (UniqueName: \"kubernetes.io/projected/2ca6195d-72ed-44c3-bc9e-449566276bfe-kube-api-access-fthrb\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825750    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-kubelet\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825928    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825946    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825964    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-ovn-node-metrics-cert\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.825993    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-ovnkube-script-lib\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826034    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-host\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826060    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c0c7485-f8c7-4a05-a496-9e8604e77b66-iptables-alerter-script\") pod \"iptables-alerter-6587w\" (UID: \"6c0c7485-f8c7-4a05-a496-9e8604e77b66\") " pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826101    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826164    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjc8q\" (UniqueName: \"kubernetes.io/projected/6c0c7485-f8c7-4a05-a496-9e8604e77b66-kube-api-access-kjc8q\") pod \"iptables-alerter-6587w\" (UID: \"6c0c7485-f8c7-4a05-a496-9e8604e77b66\") " pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826200    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-run-netns\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826241    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-run-systemd\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826268    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld5hj\" (UniqueName: \"kubernetes.io/projected/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-kube-api-access-ld5hj\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826294    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-var-lib-kubelet\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-systemd-units\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-run-openvswitch\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-modprobe-d\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-sysctl-d\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826454    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-systemd\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.826819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826494    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-tuned\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.827372 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.826863    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:38.827372 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:38.826922    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:38.827475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.827371    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 04 22:08:38.827826 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.827803    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 04 22:08:38.828094 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.828077    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 04 22:08:38.828786 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.828501    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:38.832438 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.832412    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 04 22:08:38.833605 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.832716    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pr64g\""
Sep 04 22:08:38.833605 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.832953    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:38.834000 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.833980    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 04 22:08:38.834607 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.834591    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-k8d8k\""
Sep 04 22:08:38.835659 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.835532    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 04 22:08:38.836149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.835956    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 04 22:08:38.836260 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.836241    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 04 22:08:38.836657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.836612    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 04 22:08:38.836657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.836612    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 04 22:08:38.836947 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.836930    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-n8zhp\""
Sep 04 22:08:38.837782 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.837482    2576 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 04 22:08:38.837782 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.837556    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.840783 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.840765    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:38.843469 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.843449    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:38.843573 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:38.843520    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:38.853042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.853020    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-scc45\""
Sep 04 22:08:38.854530 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.854509    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 04 22:08:38.854722 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.854705    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 04 22:08:38.855499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.855477    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 04 22:08:38.860212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.860195    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8lpx4\""
Sep 04 22:08:38.862437 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.862417    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 04 22:08:38.870849 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.870830    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 04 22:08:38.917258 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.916408    2576 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 04 22:08:38.918988 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.917795    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal" event={"ID":"3ea6adedb9dcd115be4eba74ebae07fc","Type":"ContainerStarted","Data":"dd0edb6c6d3c0eb4e6e766c128979036514d8190a93e5c50cf5010ab4fd78c0b"}
Sep 04 22:08:38.921771 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.921742    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal" event={"ID":"354628df5e5c0356cd158ffc3ea8a19c","Type":"ContainerStarted","Data":"df517c476e2dc237e6a3e2862c5b365e50014111ac8f8795b1dda593424aa32e"}
Sep 04 22:08:38.927202 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/862daba0-07cf-499c-b35f-6a0d34c49cc9-hosts-file\") pod \"node-resolver-5mt7k\" (UID: \"862daba0-07cf-499c-b35f-6a0d34c49cc9\") " pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:38.927311 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-run-k8s-cni-cncf-io\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.927311 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927233    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-modprobe-d\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.927311 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927263    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-systemd\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.927504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-tuned\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.927504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927346    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-etc-openvswitch\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.927504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-log-socket\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.927504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927398    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-modprobe-d\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.927504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927402    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-conf-dir\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.927504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-daemon-config\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.927504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-systemd\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.927504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927493    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c0c7485-f8c7-4a05-a496-9e8604e77b66-host-slash\") pod \"iptables-alerter-6587w\" (UID: \"6c0c7485-f8c7-4a05-a496-9e8604e77b66\") " pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927550    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-var-lib-openvswitch\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927577    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-run-ovn\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927604    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-node-log\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927631    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnjfn\" (UniqueName: \"kubernetes.io/projected/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-kube-api-access-xnjfn\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927748    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70230cc2-2d90-4885-be35-b21119ece277-system-cni-dir\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927776    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-run-multus-certs\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927815    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqlzx\" (UniqueName: \"kubernetes.io/projected/862daba0-07cf-499c-b35f-6a0d34c49cc9-kube-api-access-kqlzx\") pod \"node-resolver-5mt7k\" (UID: \"862daba0-07cf-499c-b35f-6a0d34c49cc9\") " pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927842    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-run\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.927870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927867    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-sys\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927952    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-var-lib-openvswitch\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927956    2576 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927984    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-sys\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928016    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-run-ovn\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928024    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-log-socket\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928054    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6c0c7485-f8c7-4a05-a496-9e8604e77b66-host-slash\") pod \"iptables-alerter-6587w\" (UID: \"6c0c7485-f8c7-4a05-a496-9e8604e77b66\") " pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928070    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-etc-openvswitch\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928177    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-node-log\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.927985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7llk\" (UniqueName: \"kubernetes.io/projected/70230cc2-2d90-4885-be35-b21119ece277-kube-api-access-j7llk\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928252    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fthrb\" (UniqueName: \"kubernetes.io/projected/2ca6195d-72ed-44c3-bc9e-449566276bfe-kube-api-access-fthrb\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928310    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-cni-binary-copy\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928346    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-run\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.928447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928358    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8add73da-6207-468b-a38f-5db9c6261331-host\") pod \"node-ca-ftrbj\" (UID: \"8add73da-6207-468b-a38f-5db9c6261331\") " pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bcq\" (UniqueName: \"kubernetes.io/projected/8add73da-6207-468b-a38f-5db9c6261331-kube-api-access-t2bcq\") pod \"node-ca-ftrbj\" (UID: \"8add73da-6207-468b-a38f-5db9c6261331\") " pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928508    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjc8q\" (UniqueName: \"kubernetes.io/projected/6c0c7485-f8c7-4a05-a496-9e8604e77b66-kube-api-access-kjc8q\") pod \"iptables-alerter-6587w\" (UID: \"6c0c7485-f8c7-4a05-a496-9e8604e77b66\") " pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-run-netns\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70230cc2-2d90-4885-be35-b21119ece277-cni-binary-copy\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-run-netns\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-sys-fs\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928675    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-cni-dir\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928694    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-cnibin\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928709    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8c26c57-2178-45fa-8855-ef16d49e2100-agent-certs\") pod \"konnectivity-agent-5d6dc\" (UID: \"b8c26c57-2178-45fa-8855-ef16d49e2100\") " pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8c26c57-2178-45fa-8855-ef16d49e2100-konnectivity-ca\") pod \"konnectivity-agent-5d6dc\" (UID: \"b8c26c57-2178-45fa-8855-ef16d49e2100\") " pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-var-lib-kubelet\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70230cc2-2d90-4885-be35-b21119ece277-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-registration-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928947    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-var-lib-kubelet\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.929003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.928959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-device-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929064    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-var-lib-cni-bin\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929106    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-sysctl-d\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929134    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-ovnkube-config\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929176    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70230cc2-2d90-4885-be35-b21119ece277-tuning-conf-dir\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-socket-dir-parent\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929234    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-sysconfig\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-kubernetes\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929283    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929308    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-slash\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929332    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-cni-bin\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929356    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-cni-netd\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929381    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-env-overrides\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929409    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-system-cni-dir\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-run-netns\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929460    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-dbus\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:38.929658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929485    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862daba0-07cf-499c-b35f-6a0d34c49cc9-tmp-dir\") pod \"node-resolver-5mt7k\" (UID: \"862daba0-07cf-499c-b35f-6a0d34c49cc9\") " pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929516    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-sysctl-conf\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929540    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-tmp\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929565    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929633    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-ovnkube-script-lib\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-lib-modules\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-kubelet\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-kubernetes\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929806    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-kubelet\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929819    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-sysctl-d\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929864    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-cni-netd\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-slash\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929890    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-sysconfig\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-ovnkube-config\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929899    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-ovn-node-metrics-cert\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929929    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-cni-bin\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.930519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929962    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-lib-modules\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.929970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70230cc2-2d90-4885-be35-b21119ece277-cnibin\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-sysctl-conf\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930047    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:38.930154    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:38.930234    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs podName:2ca6195d-72ed-44c3-bc9e-449566276bfe nodeName:}" failed. No retries permitted until 2026-09-04 22:08:39.430209277 +0000 UTC m=+2.948931814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs") pod "network-metrics-daemon-ssvlz" (UID: "2ca6195d-72ed-44c3-bc9e-449566276bfe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70230cc2-2d90-4885-be35-b21119ece277-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgd4f\" (UniqueName: \"kubernetes.io/projected/e009d639-3bbd-458f-a466-599750cf1235-kube-api-access-sgd4f\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930331    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-kubelet-config\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930372    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-host\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930403    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c0c7485-f8c7-4a05-a496-9e8604e77b66-iptables-alerter-script\") pod \"iptables-alerter-6587w\" (UID: \"6c0c7485-f8c7-4a05-a496-9e8604e77b66\") " pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930450    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-host\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930487    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-run-systemd\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930517    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld5hj\" (UniqueName: \"kubernetes.io/projected/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-kube-api-access-ld5hj\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930558    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-socket-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930586    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-var-lib-kubelet\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.931358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930614    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6vxs\" (UniqueName: \"kubernetes.io/projected/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-kube-api-access-b6vxs\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930648    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-systemd-units\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930818    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-host-run-ovn-kubernetes\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-run-openvswitch\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-systemd-units\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930894    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930943    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-run-systemd\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930959    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-var-lib-cni-multus\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930961    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6c0c7485-f8c7-4a05-a496-9e8604e77b66-iptables-alerter-script\") pod \"iptables-alerter-6587w\" (UID: \"6c0c7485-f8c7-4a05-a496-9e8604e77b66\") " pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.930984    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-run-openvswitch\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.931022    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-hostroot\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.931051    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-etc-kubernetes\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.931120    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-ovnkube-script-lib\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.931147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8add73da-6207-468b-a38f-5db9c6261331-serviceca\") pod \"node-ca-ftrbj\" (UID: \"8add73da-6207-468b-a38f-5db9c6261331\") " pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:38.932153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.931215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-env-overrides\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.932685 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.932333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-etc-tuned\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.933098 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.933074    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-tmp\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.934923 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.934890    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-ovn-node-metrics-cert\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.942350 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.942326    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnjfn\" (UniqueName: \"kubernetes.io/projected/f6d7f6b0-1e20-4e90-8aa7-4f56068a909d-kube-api-access-xnjfn\") pod \"tuned-7v6rc\" (UID: \"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d\") " pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:38.947121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.947098    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld5hj\" (UniqueName: \"kubernetes.io/projected/8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f-kube-api-access-ld5hj\") pod \"ovnkube-node-wrqjr\" (UID: \"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f\") " pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:38.948226 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.948202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjc8q\" (UniqueName: \"kubernetes.io/projected/6c0c7485-f8c7-4a05-a496-9e8604e77b66-kube-api-access-kjc8q\") pod \"iptables-alerter-6587w\" (UID: \"6c0c7485-f8c7-4a05-a496-9e8604e77b66\") " pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:38.950260 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:38.950200    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fthrb\" (UniqueName: \"kubernetes.io/projected/2ca6195d-72ed-44c3-bc9e-449566276bfe-kube-api-access-fthrb\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:39.005824 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.005786    2576 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-dkltg"
Sep 04 22:08:39.031557 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031516    2576 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dkltg"
Sep 04 22:08:39.031750 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031625    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7llk\" (UniqueName: \"kubernetes.io/projected/70230cc2-2d90-4885-be35-b21119ece277-kube-api-access-j7llk\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.031750 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031659    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:39.031750 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-cni-binary-copy\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.031750 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031708    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8add73da-6207-468b-a38f-5db9c6261331-host\") pod \"node-ca-ftrbj\" (UID: \"8add73da-6207-468b-a38f-5db9c6261331\") " pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/8add73da-6207-468b-a38f-5db9c6261331-host\") pod \"node-ca-ftrbj\" (UID: \"8add73da-6207-468b-a38f-5db9c6261331\") " pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bcq\" (UniqueName: \"kubernetes.io/projected/8add73da-6207-468b-a38f-5db9c6261331-kube-api-access-t2bcq\") pod \"node-ca-ftrbj\" (UID: \"8add73da-6207-468b-a38f-5db9c6261331\") " pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70230cc2-2d90-4885-be35-b21119ece277-cni-binary-copy\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031847    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-sys-fs\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031870    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-cni-dir\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031891    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-cnibin\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031913    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8c26c57-2178-45fa-8855-ef16d49e2100-agent-certs\") pod \"konnectivity-agent-5d6dc\" (UID: \"b8c26c57-2178-45fa-8855-ef16d49e2100\") " pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.031820    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031940    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8c26c57-2178-45fa-8855-ef16d49e2100-konnectivity-ca\") pod \"konnectivity-agent-5d6dc\" (UID: \"b8c26c57-2178-45fa-8855-ef16d49e2100\") " pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:39.031979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031966    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70230cc2-2d90-4885-be35-b21119ece277-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.031999    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret podName:6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b nodeName:}" failed. No retries permitted until 2026-09-04 22:08:39.531978379 +0000 UTC m=+3.050700906 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret") pod "global-pull-secret-syncer-8hpxw" (UID: "6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.031995    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-sys-fs\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032020    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-registration-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032048    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032076    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-device-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-var-lib-cni-bin\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032154    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70230cc2-2d90-4885-be35-b21119ece277-tuning-conf-dir\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032179    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-socket-dir-parent\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-system-cni-dir\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032250    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-run-netns\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032277    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-dbus\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032289    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-cnibin\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032305    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862daba0-07cf-499c-b35f-6a0d34c49cc9-tmp-dir\") pod \"node-resolver-5mt7k\" (UID: \"862daba0-07cf-499c-b35f-6a0d34c49cc9\") " pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70230cc2-2d90-4885-be35-b21119ece277-cnibin\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-cni-dir\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.032509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032367    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70230cc2-2d90-4885-be35-b21119ece277-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgd4f\" (UniqueName: \"kubernetes.io/projected/e009d639-3bbd-458f-a466-599750cf1235-kube-api-access-sgd4f\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032424    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-kubelet-config\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032424    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-socket-dir-parent\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032436    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/70230cc2-2d90-4885-be35-b21119ece277-cni-binary-copy\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032444    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/70230cc2-2d90-4885-be35-b21119ece277-tuning-conf-dir\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-socket-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-cni-binary-copy\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032120    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-registration-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-var-lib-kubelet\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-system-cni-dir\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032493    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-var-lib-cni-bin\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032497    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-device-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032531    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-run-netns\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032549    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6vxs\" (UniqueName: \"kubernetes.io/projected/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-kube-api-access-b6vxs\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032572    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/70230cc2-2d90-4885-be35-b21119ece277-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032580    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-kubelet-config\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:39.033040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032580    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-dbus\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032618    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-var-lib-kubelet\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-var-lib-cni-multus\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-etc-selinux\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-hostroot\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032719    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-etc-kubernetes\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032750    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/70230cc2-2d90-4885-be35-b21119ece277-cnibin\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032781    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-socket-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032791    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-etc-kubernetes\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032820    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8add73da-6207-468b-a38f-5db9c6261331-serviceca\") pod \"node-ca-ftrbj\" (UID: \"8add73da-6207-468b-a38f-5db9c6261331\") " pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032824    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-var-lib-cni-multus\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/862daba0-07cf-499c-b35f-6a0d34c49cc9-hosts-file\") pod \"node-resolver-5mt7k\" (UID: \"862daba0-07cf-499c-b35f-6a0d34c49cc9\") " pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-hostroot\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-run-k8s-cni-cncf-io\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.032964    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/862daba0-07cf-499c-b35f-6a0d34c49cc9-hosts-file\") pod \"node-resolver-5mt7k\" (UID: \"862daba0-07cf-499c-b35f-6a0d34c49cc9\") " pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033010    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-run-k8s-cni-cncf-io\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033122    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-conf-dir\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.033787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-daemon-config\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033184    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-conf-dir\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033302    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70230cc2-2d90-4885-be35-b21119ece277-system-cni-dir\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033392    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e009d639-3bbd-458f-a466-599750cf1235-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033412    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-run-multus-certs\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033429    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqlzx\" (UniqueName: \"kubernetes.io/projected/862daba0-07cf-499c-b35f-6a0d34c49cc9-kube-api-access-kqlzx\") pod \"node-resolver-5mt7k\" (UID: \"862daba0-07cf-499c-b35f-6a0d34c49cc9\") " pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033435    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/8add73da-6207-468b-a38f-5db9c6261331-serviceca\") pod \"node-ca-ftrbj\" (UID: \"8add73da-6207-468b-a38f-5db9c6261331\") " pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033458    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-host-run-multus-certs\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033471    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b8c26c57-2178-45fa-8855-ef16d49e2100-konnectivity-ca\") pod \"konnectivity-agent-5d6dc\" (UID: \"b8c26c57-2178-45fa-8855-ef16d49e2100\") " pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033518    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/70230cc2-2d90-4885-be35-b21119ece277-system-cni-dir\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033661    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/70230cc2-2d90-4885-be35-b21119ece277-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033674    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-multus-daemon-config\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.034504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.033678    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/862daba0-07cf-499c-b35f-6a0d34c49cc9-tmp-dir\") pod \"node-resolver-5mt7k\" (UID: \"862daba0-07cf-499c-b35f-6a0d34c49cc9\") " pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:39.035454 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.035433    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b8c26c57-2178-45fa-8855-ef16d49e2100-agent-certs\") pod \"konnectivity-agent-5d6dc\" (UID: \"b8c26c57-2178-45fa-8855-ef16d49e2100\") " pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:39.050343 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.050283    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:08:39.050343 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.050310    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:08:39.050343 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.050323    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ghks5 for pod openshift-network-diagnostics/network-check-target-kr6k4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:39.050568 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.050385    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5 podName:138ea0b8-99d3-4c3a-927a-0e8688d3579e nodeName:}" failed. No retries permitted until 2026-09-04 22:08:39.550369783 +0000 UTC m=+3.069092296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ghks5" (UniqueName: "kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5") pod "network-check-target-kr6k4" (UID: "138ea0b8-99d3-4c3a-927a-0e8688d3579e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:39.052289 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.052263    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6vxs\" (UniqueName: \"kubernetes.io/projected/b2f0f92e-fe37-4a40-b73e-7c7dc9186a14-kube-api-access-b6vxs\") pod \"multus-pns4q\" (UID: \"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14\") " pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.054201 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.054167    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqlzx\" (UniqueName: \"kubernetes.io/projected/862daba0-07cf-499c-b35f-6a0d34c49cc9-kube-api-access-kqlzx\") pod \"node-resolver-5mt7k\" (UID: \"862daba0-07cf-499c-b35f-6a0d34c49cc9\") " pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:39.054289 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.054226    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bcq\" (UniqueName: \"kubernetes.io/projected/8add73da-6207-468b-a38f-5db9c6261331-kube-api-access-t2bcq\") pod \"node-ca-ftrbj\" (UID: \"8add73da-6207-468b-a38f-5db9c6261331\") " pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:39.056086 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.056061    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7llk\" (UniqueName: \"kubernetes.io/projected/70230cc2-2d90-4885-be35-b21119ece277-kube-api-access-j7llk\") pod \"multus-additional-cni-plugins-slx5l\" (UID: \"70230cc2-2d90-4885-be35-b21119ece277\") " pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.056654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.056630    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgd4f\" (UniqueName: \"kubernetes.io/projected/e009d639-3bbd-458f-a466-599750cf1235-kube-api-access-sgd4f\") pod \"aws-ebs-csi-driver-node-jbjvn\" (UID: \"e009d639-3bbd-458f-a466-599750cf1235\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.071217 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.071189    2576 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 04 22:08:39.124945 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.124915    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-7v6rc"
Sep 04 22:08:39.136596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.136572    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6587w"
Sep 04 22:08:39.148232 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.148208    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:08:39.156164 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.156136    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5mt7k"
Sep 04 22:08:39.165003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.164982    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn"
Sep 04 22:08:39.171926 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.171905    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:39.177813 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.177791    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-slx5l"
Sep 04 22:08:39.182952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.182933    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pns4q"
Sep 04 22:08:39.188401 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.188383    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ftrbj"
Sep 04 22:08:39.436103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.436003    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:39.436280 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.436139    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:39.436280 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.436212    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs podName:2ca6195d-72ed-44c3-bc9e-449566276bfe nodeName:}" failed. No retries permitted until 2026-09-04 22:08:40.436192633 +0000 UTC m=+3.954915148 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs") pod "network-metrics-daemon-ssvlz" (UID: "2ca6195d-72ed-44c3-bc9e-449566276bfe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:39.536815 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.536773    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:39.537001 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.536947    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:39.537060 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.537022    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret podName:6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b nodeName:}" failed. No retries permitted until 2026-09-04 22:08:40.537002448 +0000 UTC m=+4.055724966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret") pod "global-pull-secret-syncer-8hpxw" (UID: "6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:39.637946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:39.637907    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:39.638137 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.638059    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:08:39.638137 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.638081    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:08:39.638137 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.638110    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ghks5 for pod openshift-network-diagnostics/network-check-target-kr6k4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:39.638296 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:39.638183    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5 podName:138ea0b8-99d3-4c3a-927a-0e8688d3579e nodeName:}" failed. No retries permitted until 2026-09-04 22:08:40.638161612 +0000 UTC m=+4.156884125 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghks5" (UniqueName: "kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5") pod "network-check-target-kr6k4" (UID: "138ea0b8-99d3-4c3a-927a-0e8688d3579e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:40.036265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.036231    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 22:03:39 +0000 UTC" deadline="2028-03-31 22:21:28.128344012 +0000 UTC"
Sep 04 22:08:40.036265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.036262    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13776h12m48.092086226s"
Sep 04 22:08:40.047049 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:40.046974    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6d7f6b0_1e20_4e90_8aa7_4f56068a909d.slice/crio-d800ecb658e2535ab45beb18d4dce5789d4be1e256932c024329ea1f17d14353 WatchSource:0}: Error finding container d800ecb658e2535ab45beb18d4dce5789d4be1e256932c024329ea1f17d14353: Status 404 returned error can't find the container with id d800ecb658e2535ab45beb18d4dce5789d4be1e256932c024329ea1f17d14353
Sep 04 22:08:40.048840 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:40.048754    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode009d639_3bbd_458f_a466_599750cf1235.slice/crio-f6dd89cd40fec1fc404f28f02253edf6a5c39fdab99d7e838156624345820d92 WatchSource:0}: Error finding container f6dd89cd40fec1fc404f28f02253edf6a5c39fdab99d7e838156624345820d92: Status 404 returned error can't find the container with id f6dd89cd40fec1fc404f28f02253edf6a5c39fdab99d7e838156624345820d92
Sep 04 22:08:40.050215 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:40.049989    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2f0f92e_fe37_4a40_b73e_7c7dc9186a14.slice/crio-ef05b81a0e4bb66ee4803b8de17d6a8aab68db9fc399041648dabab5308a7310 WatchSource:0}: Error finding container ef05b81a0e4bb66ee4803b8de17d6a8aab68db9fc399041648dabab5308a7310: Status 404 returned error can't find the container with id ef05b81a0e4bb66ee4803b8de17d6a8aab68db9fc399041648dabab5308a7310
Sep 04 22:08:40.052848 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:40.052735    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8103c5f8_5bcf_4b4a_a5c5_0eefc0c9013f.slice/crio-6c25c6cc32f527a1426beea2a01a0c3c534d81321d758b42e20c9620a49cf285 WatchSource:0}: Error finding container 6c25c6cc32f527a1426beea2a01a0c3c534d81321d758b42e20c9620a49cf285: Status 404 returned error can't find the container with id 6c25c6cc32f527a1426beea2a01a0c3c534d81321d758b42e20c9620a49cf285
Sep 04 22:08:40.080019 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:40.079988    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c26c57_2178_45fa_8855_ef16d49e2100.slice/crio-cf270b25da3cb5632274ebf2b2a7a961f9cf1fa45dd4a24c86ced5d73bc44aec WatchSource:0}: Error finding container cf270b25da3cb5632274ebf2b2a7a961f9cf1fa45dd4a24c86ced5d73bc44aec: Status 404 returned error can't find the container with id cf270b25da3cb5632274ebf2b2a7a961f9cf1fa45dd4a24c86ced5d73bc44aec
Sep 04 22:08:40.081226 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:40.081204    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c0c7485_f8c7_4a05_a496_9e8604e77b66.slice/crio-d48d9b71a659a05b70fb92d918c2381390844df2096770555d6c510520cfbbb7 WatchSource:0}: Error finding container d48d9b71a659a05b70fb92d918c2381390844df2096770555d6c510520cfbbb7: Status 404 returned error can't find the container with id d48d9b71a659a05b70fb92d918c2381390844df2096770555d6c510520cfbbb7
Sep 04 22:08:40.082096 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:40.082019    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8add73da_6207_468b_a38f_5db9c6261331.slice/crio-63a293bda299e8b8aec191350a109d3e82b9fd7548a84b3962fdb2e28db369d8 WatchSource:0}: Error finding container 63a293bda299e8b8aec191350a109d3e82b9fd7548a84b3962fdb2e28db369d8: Status 404 returned error can't find the container with id 63a293bda299e8b8aec191350a109d3e82b9fd7548a84b3962fdb2e28db369d8
Sep 04 22:08:40.084889 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:08:40.084446    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70230cc2_2d90_4885_be35_b21119ece277.slice/crio-fecb6058a1ed03c3dcaa29aed58614395d23a854db285d3aeddc7b7359fa77da WatchSource:0}: Error finding container fecb6058a1ed03c3dcaa29aed58614395d23a854db285d3aeddc7b7359fa77da: Status 404 returned error can't find the container with id fecb6058a1ed03c3dcaa29aed58614395d23a854db285d3aeddc7b7359fa77da
Sep 04 22:08:40.446242 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.446206    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:40.446405 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.446318    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:40.446405 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.446374    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs podName:2ca6195d-72ed-44c3-bc9e-449566276bfe nodeName:}" failed. No retries permitted until 2026-09-04 22:08:42.446358784 +0000 UTC m=+5.965081310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs") pod "network-metrics-daemon-ssvlz" (UID: "2ca6195d-72ed-44c3-bc9e-449566276bfe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:40.546970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.546919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:40.547173 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.547130    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:40.547301 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.547284    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret podName:6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b nodeName:}" failed. No retries permitted until 2026-09-04 22:08:42.547259185 +0000 UTC m=+6.065981713 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret") pod "global-pull-secret-syncer-8hpxw" (UID: "6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:40.647848 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.647740    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:40.648042 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.647954    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:08:40.648042 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.647977    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:08:40.648042 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.647991    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ghks5 for pod openshift-network-diagnostics/network-check-target-kr6k4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:40.648251 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.648060    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5 podName:138ea0b8-99d3-4c3a-927a-0e8688d3579e nodeName:}" failed. No retries permitted until 2026-09-04 22:08:42.64803638 +0000 UTC m=+6.166758898 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghks5" (UniqueName: "kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5") pod "network-check-target-kr6k4" (UID: "138ea0b8-99d3-4c3a-927a-0e8688d3579e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:40.913410 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.913258    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:40.913410 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.913401    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:40.913793 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.913750    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:40.913897 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.913865    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:40.913970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.913936    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:40.914080 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:40.914017    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:40.935004 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.934968    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftrbj" event={"ID":"8add73da-6207-468b-a38f-5db9c6261331","Type":"ContainerStarted","Data":"63a293bda299e8b8aec191350a109d3e82b9fd7548a84b3962fdb2e28db369d8"}
Sep 04 22:08:40.941887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.941800    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerStarted","Data":"fecb6058a1ed03c3dcaa29aed58614395d23a854db285d3aeddc7b7359fa77da"}
Sep 04 22:08:40.952232 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.952147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6587w" event={"ID":"6c0c7485-f8c7-4a05-a496-9e8604e77b66","Type":"ContainerStarted","Data":"d48d9b71a659a05b70fb92d918c2381390844df2096770555d6c510520cfbbb7"}
Sep 04 22:08:40.961104 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.961014    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5mt7k" event={"ID":"862daba0-07cf-499c-b35f-6a0d34c49cc9","Type":"ContainerStarted","Data":"fa0eaad0d05cc3a496e32383c70aae5cc2a5ceef4b68ec9efad6d28ab77ffb12"}
Sep 04 22:08:40.968713 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.968653    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pns4q" event={"ID":"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14","Type":"ContainerStarted","Data":"ef05b81a0e4bb66ee4803b8de17d6a8aab68db9fc399041648dabab5308a7310"}
Sep 04 22:08:40.983824 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.983718    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" event={"ID":"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f","Type":"ContainerStarted","Data":"6c25c6cc32f527a1426beea2a01a0c3c534d81321d758b42e20c9620a49cf285"}
Sep 04 22:08:40.999655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:40.999579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7v6rc" event={"ID":"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d","Type":"ContainerStarted","Data":"d800ecb658e2535ab45beb18d4dce5789d4be1e256932c024329ea1f17d14353"}
Sep 04 22:08:41.007156 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:41.006674    2576 generic.go:356] "Generic (PLEG): container finished" podID="3ea6adedb9dcd115be4eba74ebae07fc" containerID="ecb2977c8259a9c73d7901238e964fa6e681976d043021450dbaf5f5318d4d1d" exitCode=0
Sep 04 22:08:41.007156 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:41.006763    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal" event={"ID":"3ea6adedb9dcd115be4eba74ebae07fc","Type":"ContainerDied","Data":"ecb2977c8259a9c73d7901238e964fa6e681976d043021450dbaf5f5318d4d1d"}
Sep 04 22:08:41.011591 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:41.011544    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5d6dc" event={"ID":"b8c26c57-2178-45fa-8855-ef16d49e2100","Type":"ContainerStarted","Data":"cf270b25da3cb5632274ebf2b2a7a961f9cf1fa45dd4a24c86ced5d73bc44aec"}
Sep 04 22:08:41.027413 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:41.027302    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn" event={"ID":"e009d639-3bbd-458f-a466-599750cf1235","Type":"ContainerStarted","Data":"f6dd89cd40fec1fc404f28f02253edf6a5c39fdab99d7e838156624345820d92"}
Sep 04 22:08:41.033993 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:41.033961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal" event={"ID":"354628df5e5c0356cd158ffc3ea8a19c","Type":"ContainerStarted","Data":"6d39ce474386ec604249f1164da9150382a09b03f98fa2707833284b8516fe96"}
Sep 04 22:08:41.036904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:41.036839    2576 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-03 22:03:39 +0000 UTC" deadline="2028-01-29 16:22:05.882671245 +0000 UTC"
Sep 04 22:08:41.036904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:41.036871    2576 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12282h13m24.845805525s"
Sep 04 22:08:41.061882 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:41.061734    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-78.ec2.internal" podStartSLOduration=3.061713117 podStartE2EDuration="3.061713117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:41.061403279 +0000 UTC m=+4.580125816" watchObservedRunningTime="2026-09-04 22:08:41.061713117 +0000 UTC m=+4.580435657"
Sep 04 22:08:42.046759 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:42.046315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal" event={"ID":"3ea6adedb9dcd115be4eba74ebae07fc","Type":"ContainerStarted","Data":"5a79cb8cfeb9ef4a90be6bdb96df1d1fe7594d5e7d705a919b15c68beff52fcb"}
Sep 04 22:08:42.463903 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:42.463205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:42.463903 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.463383    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:42.463903 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.463453    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs podName:2ca6195d-72ed-44c3-bc9e-449566276bfe nodeName:}" failed. No retries permitted until 2026-09-04 22:08:46.463432492 +0000 UTC m=+9.982155020 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs") pod "network-metrics-daemon-ssvlz" (UID: "2ca6195d-72ed-44c3-bc9e-449566276bfe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:42.564577 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:42.564538    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:42.564775 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.564711    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:42.564847 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.564796    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret podName:6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b nodeName:}" failed. No retries permitted until 2026-09-04 22:08:46.564773722 +0000 UTC m=+10.083496253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret") pod "global-pull-secret-syncer-8hpxw" (UID: "6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:42.666309 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:42.665537    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:42.666309 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.665760    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:08:42.666309 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.665781    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:08:42.666309 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.665794    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ghks5 for pod openshift-network-diagnostics/network-check-target-kr6k4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:42.666309 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.665853    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5 podName:138ea0b8-99d3-4c3a-927a-0e8688d3579e nodeName:}" failed. No retries permitted until 2026-09-04 22:08:46.665833419 +0000 UTC m=+10.184555931 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghks5" (UniqueName: "kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5") pod "network-check-target-kr6k4" (UID: "138ea0b8-99d3-4c3a-927a-0e8688d3579e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:42.912908 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:42.912356    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:42.912908 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:42.912396    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:42.912908 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.912499    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:42.912908 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.912645    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:42.912908 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:42.912722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:42.912908 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:42.912799    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:44.912227 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:44.912187    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:44.912779 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:44.912192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:44.912779 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:44.912477    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:44.912779 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:44.912346    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:44.912779 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:44.912537    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:44.912779 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:44.912622    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:46.503743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:46.503712    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:46.504157 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.503913    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:46.504157 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.503997    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs podName:2ca6195d-72ed-44c3-bc9e-449566276bfe nodeName:}" failed. No retries permitted until 2026-09-04 22:08:54.503975034 +0000 UTC m=+18.022697554 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs") pod "network-metrics-daemon-ssvlz" (UID: "2ca6195d-72ed-44c3-bc9e-449566276bfe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:46.604994 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:46.604956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:46.605193 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.605172    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:46.605278 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.605241    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret podName:6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b nodeName:}" failed. No retries permitted until 2026-09-04 22:08:54.605221702 +0000 UTC m=+18.123944236 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret") pod "global-pull-secret-syncer-8hpxw" (UID: "6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:46.705547 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:46.705514    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:46.705721 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.705684    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:08:46.705721 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.705704    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:08:46.705721 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.705717    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ghks5 for pod openshift-network-diagnostics/network-check-target-kr6k4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:46.705896 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.705776    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5 podName:138ea0b8-99d3-4c3a-927a-0e8688d3579e nodeName:}" failed. No retries permitted until 2026-09-04 22:08:54.705757003 +0000 UTC m=+18.224479516 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghks5" (UniqueName: "kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5") pod "network-check-target-kr6k4" (UID: "138ea0b8-99d3-4c3a-927a-0e8688d3579e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:46.912967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:46.912882    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:46.913140 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:46.912984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:46.913140 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.913104    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:46.913478 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.913453    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:46.913554 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:46.913505    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:46.913612 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:46.913589    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:48.912929 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:48.912772    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:48.912929 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:48.912803    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:48.912929 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:48.912779    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:48.912929 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:48.912893    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:48.913592 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:48.913007    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:48.913592 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:48.913118    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:50.912454 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:50.912426    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:50.912848 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:50.912431    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:50.912848 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:50.912549    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:50.912848 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:50.912546    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:50.912848 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:50.912660    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:50.912848 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:50.912722    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:52.912724 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:52.912494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:52.913172 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:52.912494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:52.913172 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:52.912494    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:52.913172 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:52.912853    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:52.913172 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:52.912922    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:52.913172 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:52.912991    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:54.565380 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:54.565338    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:54.565777 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.565449    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:54.565777 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.565501    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs podName:2ca6195d-72ed-44c3-bc9e-449566276bfe nodeName:}" failed. No retries permitted until 2026-09-04 22:09:10.565485861 +0000 UTC m=+34.084208371 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs") pod "network-metrics-daemon-ssvlz" (UID: "2ca6195d-72ed-44c3-bc9e-449566276bfe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:08:54.666200 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:54.666159    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:54.666344 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.666308    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:54.666379 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.666374    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret podName:6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b nodeName:}" failed. No retries permitted until 2026-09-04 22:09:10.666360085 +0000 UTC m=+34.185082602 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret") pod "global-pull-secret-syncer-8hpxw" (UID: "6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:08:54.767373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:54.767334    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:54.767553 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.767527    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:08:54.767621 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.767559    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:08:54.767621 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.767574    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ghks5 for pod openshift-network-diagnostics/network-check-target-kr6k4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:54.767712 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.767643    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5 podName:138ea0b8-99d3-4c3a-927a-0e8688d3579e nodeName:}" failed. No retries permitted until 2026-09-04 22:09:10.767622986 +0000 UTC m=+34.286345511 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghks5" (UniqueName: "kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5") pod "network-check-target-kr6k4" (UID: "138ea0b8-99d3-4c3a-927a-0e8688d3579e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:08:54.912760 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:54.912656    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:54.912760 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:54.912690    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:54.912952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:54.912659    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:54.912952 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.912799    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:54.912952 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.912862    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:54.912952 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:54.912929    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:56.913536 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:56.913500    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:56.913965 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:56.913636    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:56.913965 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:56.913686    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:56.913965 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:56.913778    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:56.913965 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:56.913827    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:56.913965 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:56.913885    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:58.912961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:58.912738    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:08:58.913740 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:58.912757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:08:58.913740 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:58.912798    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:08:58.913740 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:58.913093    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:08:58.913740 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:58.913223    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:08:58.913740 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:08:58.913287    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:08:59.084851 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.084811    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ftrbj" event={"ID":"8add73da-6207-468b-a38f-5db9c6261331","Type":"ContainerStarted","Data":"7a55801d2980f5d8f6e9d28ce6ff6271efd56bad3a09cd8b931c3d75da3d4f9f"}
Sep 04 22:08:59.086737 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.086709    2576 generic.go:356] "Generic (PLEG): container finished" podID="70230cc2-2d90-4885-be35-b21119ece277" containerID="2b2f0e32a6d277e7c518e6a69504c9c8f7ac0c7b414faf9201cfade650dbbfc4" exitCode=0
Sep 04 22:08:59.086892 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.086798    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerDied","Data":"2b2f0e32a6d277e7c518e6a69504c9c8f7ac0c7b414faf9201cfade650dbbfc4"}
Sep 04 22:08:59.088504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.088485    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5mt7k" event={"ID":"862daba0-07cf-499c-b35f-6a0d34c49cc9","Type":"ContainerStarted","Data":"e6b33f8348729c1d9480ddd2744db3da51b440bafec733a98fafe8466f55a371"}
Sep 04 22:08:59.090104 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.090077    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pns4q" event={"ID":"b2f0f92e-fe37-4a40-b73e-7c7dc9186a14","Type":"ContainerStarted","Data":"2dada3bf5bbbc4651dcef8537295f953f0a241127c1290d73e4c27b001b478a2"}
Sep 04 22:08:59.094026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.093999    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" event={"ID":"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f","Type":"ContainerStarted","Data":"ec56c22e5a3970828a5afb6657e2127f7f0adf55f1f2a0ea71abfd0219a3eca1"}
Sep 04 22:08:59.094149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.094092    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" event={"ID":"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f","Type":"ContainerStarted","Data":"7de196d096ac114ced35976a8051d9316273e382196273094e2d77018ac42652"}
Sep 04 22:08:59.094149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.094128    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" event={"ID":"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f","Type":"ContainerStarted","Data":"abde5217bddb2e89d00cbee52defda244163103f995552a5018e9015087f97a5"}
Sep 04 22:08:59.094226 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.094170    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" event={"ID":"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f","Type":"ContainerStarted","Data":"b65ba9a628241dbd6298bf01a655dfe2d74c7a4077105817d505b8498b39ee9d"}
Sep 04 22:08:59.094226 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.094211    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" event={"ID":"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f","Type":"ContainerStarted","Data":"c811e7033df1c310b5c6eb1a87a89384e7e06c0a16fec2a22215bab5eae8a14a"}
Sep 04 22:08:59.094304 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.094242    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" event={"ID":"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f","Type":"ContainerStarted","Data":"2e60aa8a1e339df8fc0dc2e14a1880ae7451d64f20963fcf001590fa3bf258e0"}
Sep 04 22:08:59.095578 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.095553    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-7v6rc" event={"ID":"f6d7f6b0-1e20-4e90-8aa7-4f56068a909d","Type":"ContainerStarted","Data":"f3d207a88d3435c313044909689d073257825e6b9ec62f2ef50fa991756bbb1a"}
Sep 04 22:08:59.097152 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.097098    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5d6dc" event={"ID":"b8c26c57-2178-45fa-8855-ef16d49e2100","Type":"ContainerStarted","Data":"5dd7584647391ef20c28b5945dd7bce821655496a8dfe182fae5b29a66b97008"}
Sep 04 22:08:59.098699 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.098679    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn" event={"ID":"e009d639-3bbd-458f-a466-599750cf1235","Type":"ContainerStarted","Data":"50505f7d92fe27ead896d00738353d27b56de054db4fbfed752117da517b7485"}
Sep 04 22:08:59.103731 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.103692    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ftrbj" podStartSLOduration=4.036334866 podStartE2EDuration="22.10367706s" totalImagesPullingTime="18.067342194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:59.103390333 +0000 UTC m=+22.622112879" watchObservedRunningTime="2026-09-04 22:08:59.10367706 +0000 UTC m=+22.622399607"
Sep 04 22:08:59.104198 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.104167    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-78.ec2.internal" podStartSLOduration=21.104157204 podStartE2EDuration="21.104157204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:42.072806935 +0000 UTC m=+5.591529470" watchObservedRunningTime="2026-09-04 22:08:59.104157204 +0000 UTC m=+22.622879737"
Sep 04 22:08:59.121324 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.121277    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5d6dc" podStartSLOduration=11.32548028 podStartE2EDuration="22.121263388s" totalImagesPullingTime="10.795783108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:59.121248168 +0000 UTC m=+22.639970703" watchObservedRunningTime="2026-09-04 22:08:59.121263388 +0000 UTC m=+22.639985916"
Sep 04 22:08:59.141581 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.141516    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pns4q" podStartSLOduration=3.989331182 podStartE2EDuration="22.14149608s" totalImagesPullingTime="18.152164898s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:59.141147581 +0000 UTC m=+22.659870117" watchObservedRunningTime="2026-09-04 22:08:59.14149608 +0000 UTC m=+22.660218614"
Sep 04 22:08:59.165948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.165894    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-7v6rc" podStartSLOduration=4.060240804 podStartE2EDuration="22.165877403s" totalImagesPullingTime="18.105636599s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:59.165366988 +0000 UTC m=+22.684089520" watchObservedRunningTime="2026-09-04 22:08:59.165877403 +0000 UTC m=+22.684599935"
Sep 04 22:08:59.212211 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.212156    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5mt7k" podStartSLOduration=4.137271917 podStartE2EDuration="22.21214025s" totalImagesPullingTime="18.074868333s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:08:59.211635708 +0000 UTC m=+22.730358238" watchObservedRunningTime="2026-09-04 22:08:59.21214025 +0000 UTC m=+22.730862783"
Sep 04 22:08:59.316242 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.316209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:08:59.395645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.395588    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:08:59.581823 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.581678    2576 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 04 22:08:59.778618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.778580    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:59.779166 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.779150    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:08:59.878810 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.878714    2576 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-04T22:08:59.58181953Z","UUID":"3ff70072-172f-4f33-9e64-854212a56f01","Handler":null,"Name":"","Endpoint":""}
Sep 04 22:08:59.880118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.880101    2576 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 04 22:08:59.880200 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:08:59.880130    2576 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 04 22:09:00.102554 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:00.102474    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6587w" event={"ID":"6c0c7485-f8c7-4a05-a496-9e8604e77b66","Type":"ContainerStarted","Data":"d6267cfd0afc56f4010e56e95b50a4694546ae7798a18cf48e258802bdae1a37"}
Sep 04 22:09:00.104727 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:00.104685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn" event={"ID":"e009d639-3bbd-458f-a466-599750cf1235","Type":"ContainerStarted","Data":"77599be670ff3c06eee0dbdab0ebff59e2117fc3c629ff6b5ce17e1a71eb9859"}
Sep 04 22:09:00.308706 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:00.308645    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:00.428796 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:00.428718    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:00.912765 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:00.912739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:09:00.912878 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:00.912739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:00.912878 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:00.912851    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:00.912878 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:00.912852    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:09:00.913046 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:00.912917    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:09:00.913046 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:00.912983    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:09:01.109355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:01.109314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn" event={"ID":"e009d639-3bbd-458f-a466-599750cf1235","Type":"ContainerStarted","Data":"dcf5bcbbf47e3374b6a7efdf233d32ac89d158c996c9cbb2d7a01e758e2b6fd5"}
Sep 04 22:09:01.113255 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:01.113215    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" event={"ID":"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f","Type":"ContainerStarted","Data":"44f482b676b2f6648272dc12811df9972490ab7776df77f17f6c75814bbc89cd"}
Sep 04 22:09:01.113438 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:01.113421    2576 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 04 22:09:01.134644 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:01.134595    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6587w" podStartSLOduration=6.0674653020000004 podStartE2EDuration="24.134578187s" totalImagesPullingTime="18.067112885s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:09:00.122883969 +0000 UTC m=+23.641606504" watchObservedRunningTime="2026-09-04 22:09:01.134578187 +0000 UTC m=+24.653300721"
Sep 04 22:09:01.134871 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:01.134848    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jbjvn" podStartSLOduration=3.897276958 podStartE2EDuration="24.134842661s" totalImagesPullingTime="20.237565703s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:09:01.133290341 +0000 UTC m=+24.652012875" watchObservedRunningTime="2026-09-04 22:09:01.134842661 +0000 UTC m=+24.653565194"
Sep 04 22:09:01.507922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:01.507876    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:09:01.508466 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:01.508449    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5d6dc"
Sep 04 22:09:01.582391 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:01.582354    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:02.912363 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:02.912329    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:09:02.912363 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:02.912314    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:02.912898 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:02.912370    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:02.912898 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:02.912474    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:09:02.912898 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:02.912565    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:09:02.912898 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:02.912635    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:09:04.912274 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:04.912048    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:04.912675 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:04.912072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:04.912675 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:04.912307    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:09:04.912675 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:04.912125    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:09:04.912675 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:04.912408    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:09:04.912675 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:04.912466    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:09:05.125660 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:05.125624    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerStarted","Data":"c8a5fbcc7ad25ecb6af5792412795510a217ec993a8bf6adf0984a7f8db1d75e"}
Sep 04 22:09:05.129616 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:05.129574    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" event={"ID":"8103c5f8-5bcf-4b4a-a5c5-0eefc0c9013f","Type":"ContainerStarted","Data":"0ad2abc65af182bc4872a50385572032c397928161c91e52cdd8e83732e3c326"}
Sep 04 22:09:05.129848 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:05.129827    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:09:05.129915 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:05.129893    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:09:05.145483 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:05.145461    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:09:05.192484 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:05.192395    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr" podStartSLOduration=10.054052199 podStartE2EDuration="28.192378547s" totalImagesPullingTime="18.138326348s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:09:05.190471367 +0000 UTC m=+28.709193890" watchObservedRunningTime="2026-09-04 22:09:05.192378547 +0000 UTC m=+28.711101074"
Sep 04 22:09:06.133983 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.133943    2576 generic.go:356] "Generic (PLEG): container finished" podID="70230cc2-2d90-4885-be35-b21119ece277" containerID="c8a5fbcc7ad25ecb6af5792412795510a217ec993a8bf6adf0984a7f8db1d75e" exitCode=0
Sep 04 22:09:06.134638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.133995    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerDied","Data":"c8a5fbcc7ad25ecb6af5792412795510a217ec993a8bf6adf0984a7f8db1d75e"}
Sep 04 22:09:06.134638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.134495    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:09:06.150819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.150776    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:09:06.691845 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.691808    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kr6k4"]
Sep 04 22:09:06.692038 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.691961    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:06.692129 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:06.692067    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:09:06.694652 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.694624    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8hpxw"]
Sep 04 22:09:06.694796 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.694735    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:06.694869 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:06.694846    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:09:06.695229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.695208    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ssvlz"]
Sep 04 22:09:06.695326 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:06.695317    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:09:06.695435 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:06.695418    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:09:07.912588 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:07.912409    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:07.913021 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:07.912666    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:09:07.999350 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:07.999309    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:08.110841 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:08.110803    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:08.140309 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:08.140277    2576 generic.go:356] "Generic (PLEG): container finished" podID="70230cc2-2d90-4885-be35-b21119ece277" containerID="2ee86e7d02663ed0b89a0dd01dfbd4a5b51232ceb2e7460e5478e7d499719c78" exitCode=0
Sep 04 22:09:08.140473 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:08.140358    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerDied","Data":"2ee86e7d02663ed0b89a0dd01dfbd4a5b51232ceb2e7460e5478e7d499719c78"}
Sep 04 22:09:08.235785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:08.235752    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:08.311445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:08.311404    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:08.912288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:08.912257    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:08.912464 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:08.912424    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-8hpxw" podUID="6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b"
Sep 04 22:09:08.912655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:08.912623    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:09:08.913098 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:08.912762    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ssvlz" podUID="2ca6195d-72ed-44c3-bc9e-449566276bfe"
Sep 04 22:09:09.037193 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:09.037110    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:09.141779 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:09.141739    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:09.912299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:09.912268    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:09.912469 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:09.912369    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kr6k4" podUID="138ea0b8-99d3-4c3a-927a-0e8688d3579e"
Sep 04 22:09:10.146626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.146593    2576 generic.go:356] "Generic (PLEG): container finished" podID="70230cc2-2d90-4885-be35-b21119ece277" containerID="8416f60b3cfae5b7e0c52a8ef07ca501e15f2bc953cfd6d5124be48811c7d7f2" exitCode=0
Sep 04 22:09:10.147018 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.146642    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerDied","Data":"8416f60b3cfae5b7e0c52a8ef07ca501e15f2bc953cfd6d5124be48811c7d7f2"}
Sep 04 22:09:10.575283 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.575255    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:09:10.575432 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:10.575354    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:09:10.575432 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:10.575402    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs podName:2ca6195d-72ed-44c3-bc9e-449566276bfe nodeName:}" failed. No retries permitted until 2026-09-04 22:09:42.575389827 +0000 UTC m=+66.094112338 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs") pod "network-metrics-daemon-ssvlz" (UID: "2ca6195d-72ed-44c3-bc9e-449566276bfe") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 04 22:09:10.676205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.676180    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:10.676336 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:10.676317    2576 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 04 22:09:10.676391 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:10.676381    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret podName:6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b nodeName:}" failed. No retries permitted until 2026-09-04 22:09:42.676366147 +0000 UTC m=+66.195088662 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret") pod "global-pull-secret-syncer-8hpxw" (UID: "6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b") : object "kube-system"/"original-pull-secret" not registered
Sep 04 22:09:10.767214 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.767184    2576 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-78.ec2.internal" event="NodeReady"
Sep 04 22:09:10.767386 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.767331    2576 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 04 22:09:10.776687 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.776657    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:10.776825 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:10.776812    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 04 22:09:10.776868 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:10.776830    2576 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 04 22:09:10.776868 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:10.776842    2576 projected.go:196] Error preparing data for projected volume kube-api-access-ghks5 for pod openshift-network-diagnostics/network-check-target-kr6k4: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:09:10.776941 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:10.776900    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5 podName:138ea0b8-99d3-4c3a-927a-0e8688d3579e nodeName:}" failed. No retries permitted until 2026-09-04 22:09:42.776882757 +0000 UTC m=+66.295605283 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ghks5" (UniqueName: "kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5") pod "network-check-target-kr6k4" (UID: "138ea0b8-99d3-4c3a-927a-0e8688d3579e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 04 22:09:10.823061 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.823014    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-79rjk"]
Sep 04 22:09:10.834384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.834314    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-76mz2"]
Sep 04 22:09:10.834538 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.834437    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:10.838439 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.838410    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 04 22:09:10.838439 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.838423    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 04 22:09:10.838625 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.838591    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wcrtk\""
Sep 04 22:09:10.844316 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.844293    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-79rjk"]
Sep 04 22:09:10.844426 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.844405    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-76mz2"
Sep 04 22:09:10.844477 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.844423    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-76mz2"]
Sep 04 22:09:10.847322 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.847141    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 04 22:09:10.847565 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.847547    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-psbpz\""
Sep 04 22:09:10.847669 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.847569    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 04 22:09:10.847859 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.847839    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 04 22:09:10.911800 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.911757    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:09:10.911973 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.911954    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:10.914709 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.914684    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 04 22:09:10.914709 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.914698    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c64pf\""
Sep 04 22:09:10.914908 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.914842    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 04 22:09:10.958112 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.958069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:10.978069 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.978027    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2faabeaa-73ea-4a77-b707-765541109e4d-config-volume\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:10.978185 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.978079    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6434544-e7c3-4235-850c-3d4471c2c030-cert\") pod \"ingress-canary-76mz2\" (UID: \"f6434544-e7c3-4235-850c-3d4471c2c030\") " pod="openshift-ingress-canary/ingress-canary-76mz2"
Sep 04 22:09:10.978185 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.978097    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwdd\" (UniqueName: \"kubernetes.io/projected/f6434544-e7c3-4235-850c-3d4471c2c030-kube-api-access-tqwdd\") pod \"ingress-canary-76mz2\" (UID: \"f6434544-e7c3-4235-850c-3d4471c2c030\") " pod="openshift-ingress-canary/ingress-canary-76mz2"
Sep 04 22:09:10.978185 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.978117    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2faabeaa-73ea-4a77-b707-765541109e4d-tmp-dir\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:10.978185 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.978147    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:10.978185 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:10.978161    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w88n\" (UniqueName: \"kubernetes.io/projected/2faabeaa-73ea-4a77-b707-765541109e4d-kube-api-access-7w88n\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:11.042199 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.042127    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:11.078720 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.078684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2faabeaa-73ea-4a77-b707-765541109e4d-config-volume\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:11.078885 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.078743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6434544-e7c3-4235-850c-3d4471c2c030-cert\") pod \"ingress-canary-76mz2\" (UID: \"f6434544-e7c3-4235-850c-3d4471c2c030\") " pod="openshift-ingress-canary/ingress-canary-76mz2"
Sep 04 22:09:11.078961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.078888    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwdd\" (UniqueName: \"kubernetes.io/projected/f6434544-e7c3-4235-850c-3d4471c2c030-kube-api-access-tqwdd\") pod \"ingress-canary-76mz2\" (UID: \"f6434544-e7c3-4235-850c-3d4471c2c030\") " pod="openshift-ingress-canary/ingress-canary-76mz2"
Sep 04 22:09:11.078961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.078941    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2faabeaa-73ea-4a77-b707-765541109e4d-tmp-dir\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:11.079082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.078981    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:11.079082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.079007    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w88n\" (UniqueName: \"kubernetes.io/projected/2faabeaa-73ea-4a77-b707-765541109e4d-kube-api-access-7w88n\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:11.079340 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:11.079318    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:09:11.079417 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.079339    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2faabeaa-73ea-4a77-b707-765541109e4d-config-volume\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:11.079417 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:11.079389    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls podName:2faabeaa-73ea-4a77-b707-765541109e4d nodeName:}" failed. No retries permitted until 2026-09-04 22:09:11.579369848 +0000 UTC m=+35.098092360 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls") pod "dns-default-79rjk" (UID: "2faabeaa-73ea-4a77-b707-765541109e4d") : secret "dns-default-metrics-tls" not found
Sep 04 22:09:11.079567 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.079549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2faabeaa-73ea-4a77-b707-765541109e4d-tmp-dir\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:11.083585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.083461    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f6434544-e7c3-4235-850c-3d4471c2c030-cert\") pod \"ingress-canary-76mz2\" (UID: \"f6434544-e7c3-4235-850c-3d4471c2c030\") " pod="openshift-ingress-canary/ingress-canary-76mz2"
Sep 04 22:09:11.091579 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.091524    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w88n\" (UniqueName: \"kubernetes.io/projected/2faabeaa-73ea-4a77-b707-765541109e4d-kube-api-access-7w88n\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:11.091697 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.091642    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwdd\" (UniqueName: \"kubernetes.io/projected/f6434544-e7c3-4235-850c-3d4471c2c030-kube-api-access-tqwdd\") pod \"ingress-canary-76mz2\" (UID: \"f6434544-e7c3-4235-850c-3d4471c2c030\") " pod="openshift-ingress-canary/ingress-canary-76mz2"
Sep 04 22:09:11.155230 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.155200    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-76mz2"
Sep 04 22:09:11.487518 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.487481    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-76mz2"]
Sep 04 22:09:11.493419 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:09:11.493392    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6434544_e7c3_4235_850c_3d4471c2c030.slice/crio-957fe3e6f7aa18e0c59967d4c35651a7967f1d18a52b2049ffa3e56a4adc21d7 WatchSource:0}: Error finding container 957fe3e6f7aa18e0c59967d4c35651a7967f1d18a52b2049ffa3e56a4adc21d7: Status 404 returned error can't find the container with id 957fe3e6f7aa18e0c59967d4c35651a7967f1d18a52b2049ffa3e56a4adc21d7
Sep 04 22:09:11.582642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.582610    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:11.582803 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:11.582769    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:09:11.582856 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:11.582848    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls podName:2faabeaa-73ea-4a77-b707-765541109e4d nodeName:}" failed. No retries permitted until 2026-09-04 22:09:12.582825684 +0000 UTC m=+36.101548215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls") pod "dns-default-79rjk" (UID: "2faabeaa-73ea-4a77-b707-765541109e4d") : secret "dns-default-metrics-tls" not found
Sep 04 22:09:11.912351 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.912275    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:11.915850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.915827    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 04 22:09:11.915974 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.915898    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 04 22:09:11.916887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:11.916869    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5pc9f\""
Sep 04 22:09:12.154042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:12.153995    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-76mz2" event={"ID":"f6434544-e7c3-4235-850c-3d4471c2c030","Type":"ContainerStarted","Data":"957fe3e6f7aa18e0c59967d4c35651a7967f1d18a52b2049ffa3e56a4adc21d7"}
Sep 04 22:09:12.590822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:12.590783    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:12.591665 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:12.590917    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:09:12.591665 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:12.590973    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls podName:2faabeaa-73ea-4a77-b707-765541109e4d nodeName:}" failed. No retries permitted until 2026-09-04 22:09:14.590957178 +0000 UTC m=+38.109679691 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls") pod "dns-default-79rjk" (UID: "2faabeaa-73ea-4a77-b707-765541109e4d") : secret "dns-default-metrics-tls" not found
Sep 04 22:09:14.607915 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:14.607865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:14.608362 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:14.608036    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:09:14.608362 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:14.608117    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls podName:2faabeaa-73ea-4a77-b707-765541109e4d nodeName:}" failed. No retries permitted until 2026-09-04 22:09:18.608094583 +0000 UTC m=+42.126817103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls") pod "dns-default-79rjk" (UID: "2faabeaa-73ea-4a77-b707-765541109e4d") : secret "dns-default-metrics-tls" not found
Sep 04 22:09:16.166429 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:16.166041    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-76mz2" event={"ID":"f6434544-e7c3-4235-850c-3d4471c2c030","Type":"ContainerStarted","Data":"4e6c3a8675e497753b5a681f07d43ad28e5ecf710380102eb206394eb1014508"}
Sep 04 22:09:16.169600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:16.169569    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerStarted","Data":"d57e80b8e9d8b396a0c506dc5bed674ab8bc7f3afd443babcb4b5c4396561a20"}
Sep 04 22:09:16.189882 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:16.188163    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-76mz2" podStartSLOduration=1.790582356 podStartE2EDuration="6.188145259s" totalImagesPullingTime="4.397562903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:09:16.187447435 +0000 UTC m=+39.706169968" watchObservedRunningTime="2026-09-04 22:09:16.188145259 +0000 UTC m=+39.706867791"
Sep 04 22:09:16.428340 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:16.428261    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:16.505877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:16.505830    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:17.175227 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:17.175199    2576 generic.go:356] "Generic (PLEG): container finished" podID="70230cc2-2d90-4885-be35-b21119ece277" containerID="d57e80b8e9d8b396a0c506dc5bed674ab8bc7f3afd443babcb4b5c4396561a20" exitCode=0
Sep 04 22:09:17.175227 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:17.175222    2576 generic.go:356] "Generic (PLEG): container finished" podID="70230cc2-2d90-4885-be35-b21119ece277" containerID="ac3f6d7c3556785c0804741b6751a1d89ff57982bc9db0ac7d49f95a0ba160d7" exitCode=0
Sep 04 22:09:17.175648 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:17.175269    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerDied","Data":"d57e80b8e9d8b396a0c506dc5bed674ab8bc7f3afd443babcb4b5c4396561a20"}
Sep 04 22:09:17.175648 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:17.175329    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerDied","Data":"ac3f6d7c3556785c0804741b6751a1d89ff57982bc9db0ac7d49f95a0ba160d7"}
Sep 04 22:09:17.187516 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:17.187485    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:17.262022 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:17.261990    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:18.180853 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:18.180816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-slx5l" event={"ID":"70230cc2-2d90-4885-be35-b21119ece277","Type":"ContainerStarted","Data":"c368f1e9a5052af03313227d939c39d6028c0e218d52e1a83f5bda1335da65a5"}
Sep 04 22:09:18.206998 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:18.206952    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-slx5l" podStartSLOduration=9.159858899 podStartE2EDuration="41.20693782s" totalImagesPullingTime="32.047078921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:09:18.205534881 +0000 UTC m=+41.724257413" watchObservedRunningTime="2026-09-04 22:09:18.20693782 +0000 UTC m=+41.725660353"
Sep 04 22:09:18.331458 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:18.331425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:18.432420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:18.432334    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:18.638175 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:18.638133    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:18.638308 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:18.638252    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:09:18.638353 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:18.638322    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls podName:2faabeaa-73ea-4a77-b707-765541109e4d nodeName:}" failed. No retries permitted until 2026-09-04 22:09:26.638302964 +0000 UTC m=+50.157025490 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls") pod "dns-default-79rjk" (UID: "2faabeaa-73ea-4a77-b707-765541109e4d") : secret "dns-default-metrics-tls" not found
Sep 04 22:09:21.219513 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:21.219478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:22.716361 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:22.716326    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:25.715287 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:25.715248    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:25.778499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:25.778466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:26.694307 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:26.694257    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:26.694500 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:26.694390    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:09:26.694500 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:26.694458    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls podName:2faabeaa-73ea-4a77-b707-765541109e4d nodeName:}" failed. No retries permitted until 2026-09-04 22:09:42.694441868 +0000 UTC m=+66.213164379 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls") pod "dns-default-79rjk" (UID: "2faabeaa-73ea-4a77-b707-765541109e4d") : secret "dns-default-metrics-tls" not found
Sep 04 22:09:38.151151 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:38.151123    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-wrqjr"
Sep 04 22:09:38.354506 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:38.354472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:38.473410 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:38.473377    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:42.605730 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.605677    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:09:42.608649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.608631    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 04 22:09:42.616119 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:42.616099    2576 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 04 22:09:42.616183 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:42.616172    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs podName:2ca6195d-72ed-44c3-bc9e-449566276bfe nodeName:}" failed. No retries permitted until 2026-09-04 22:10:46.61615221 +0000 UTC m=+130.134874723 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs") pod "network-metrics-daemon-ssvlz" (UID: "2ca6195d-72ed-44c3-bc9e-449566276bfe") : secret "metrics-daemon-secret" not found
Sep 04 22:09:42.706463 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.706430    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:42.706555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.706468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:09:42.706606 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:42.706569    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:09:42.706641 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:09:42.706626    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls podName:2faabeaa-73ea-4a77-b707-765541109e4d nodeName:}" failed. No retries permitted until 2026-09-04 22:10:14.706613242 +0000 UTC m=+98.225335752 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls") pod "dns-default-79rjk" (UID: "2faabeaa-73ea-4a77-b707-765541109e4d") : secret "dns-default-metrics-tls" not found
Sep 04 22:09:42.709278 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.709263    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 04 22:09:42.719740 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.719718    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b-original-pull-secret\") pod \"global-pull-secret-syncer-8hpxw\" (UID: \"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b\") " pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:42.729896 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.729867    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-8hpxw"
Sep 04 22:09:42.807677 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.807644    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:42.810665 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.810648    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 04 22:09:42.820624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.820606    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 04 22:09:42.830117 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:42.830086    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghks5\" (UniqueName: \"kubernetes.io/projected/138ea0b8-99d3-4c3a-927a-0e8688d3579e-kube-api-access-ghks5\") pod \"network-check-target-kr6k4\" (UID: \"138ea0b8-99d3-4c3a-927a-0e8688d3579e\") " pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:43.003850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:43.003813    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-8hpxw"]
Sep 04 22:09:43.006737 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:09:43.006693    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbcb5c1_ac22_4f52_b3b0_fb3c28a5fc9b.slice/crio-23a440a983f1da7d78c1211e59b74b0f1f29b62b98d9966a077526086e88aec4 WatchSource:0}: Error finding container 23a440a983f1da7d78c1211e59b74b0f1f29b62b98d9966a077526086e88aec4: Status 404 returned error can't find the container with id 23a440a983f1da7d78c1211e59b74b0f1f29b62b98d9966a077526086e88aec4
Sep 04 22:09:43.123910 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:43.123879    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-5pc9f\""
Sep 04 22:09:43.131705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:43.131689    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:43.241674 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:43.241644    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8hpxw" event={"ID":"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b","Type":"ContainerStarted","Data":"23a440a983f1da7d78c1211e59b74b0f1f29b62b98d9966a077526086e88aec4"}
Sep 04 22:09:43.408994 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:43.408909    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kr6k4"]
Sep 04 22:09:43.412198 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:09:43.412167    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod138ea0b8_99d3_4c3a_927a_0e8688d3579e.slice/crio-3f4e5e0e69c6663754b9281dd0134ad3f260f2c736c99325b2fe47cfb3120efc WatchSource:0}: Error finding container 3f4e5e0e69c6663754b9281dd0134ad3f260f2c736c99325b2fe47cfb3120efc: Status 404 returned error can't find the container with id 3f4e5e0e69c6663754b9281dd0134ad3f260f2c736c99325b2fe47cfb3120efc
Sep 04 22:09:44.246154 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:44.246111    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kr6k4" event={"ID":"138ea0b8-99d3-4c3a-927a-0e8688d3579e","Type":"ContainerStarted","Data":"3f4e5e0e69c6663754b9281dd0134ad3f260f2c736c99325b2fe47cfb3120efc"}
Sep 04 22:09:46.094685 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:46.094646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:46.170781 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:46.170740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:09:48.258639 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:48.258592    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kr6k4" event={"ID":"138ea0b8-99d3-4c3a-927a-0e8688d3579e","Type":"ContainerStarted","Data":"3606afb983a1232135c1d862d77f535dbbeadc9ba5bbd52a386f1ec2a7a4c85e"}
Sep 04 22:09:48.259111 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:48.258659    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:09:48.260039 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:48.260013    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-8hpxw" event={"ID":"6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b","Type":"ContainerStarted","Data":"762db46842c1c3dd682c13e87803edfc8fa4d96d8e2599faa15664dba148a6ce"}
Sep 04 22:09:48.291806 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:48.291759    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-8hpxw" podStartSLOduration=65.501197875 podStartE2EDuration="1m10.291745081s" totalImagesPullingTime="4.790547206s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:08:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:09:48.291167805 +0000 UTC m=+71.809890339" watchObservedRunningTime="2026-09-04 22:09:48.291745081 +0000 UTC m=+71.810467631"
Sep 04 22:09:48.291924 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:09:48.291880    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kr6k4" podStartSLOduration=66.910895185 podStartE2EDuration="1m11.291874924s" totalImagesPullingTime="4.380979739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:09:48.275109945 +0000 UTC m=+71.793832477" watchObservedRunningTime="2026-09-04 22:09:48.291874924 +0000 UTC m=+71.810597448"
Sep 04 22:10:00.259206 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.259100    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-58wlk"]
Sep 04 22:10:00.261391 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.261373    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.264067 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.264019    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 04 22:10:00.264195 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.264151    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 04 22:10:00.264195 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.264170    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 04 22:10:00.265211 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.265195    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-f2csf\""
Sep 04 22:10:00.265211 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.265204    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 04 22:10:00.269417 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.269400    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 04 22:10:00.273015 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.272992    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-58wlk"]
Sep 04 22:10:00.324466 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.324430    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85c7d188-f68c-4717-884d-fbf00586d73d-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.324466 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.324469    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85c7d188-f68c-4717-884d-fbf00586d73d-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.324692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.324499    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85c7d188-f68c-4717-884d-fbf00586d73d-tmp\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.324692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.324517    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85c7d188-f68c-4717-884d-fbf00586d73d-snapshots\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.324692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.324532    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85c7d188-f68c-4717-884d-fbf00586d73d-serving-cert\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.324692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.324550    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tznfg\" (UniqueName: \"kubernetes.io/projected/85c7d188-f68c-4717-884d-fbf00586d73d-kube-api-access-tznfg\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.378036 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.377998    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"]
Sep 04 22:10:00.380021 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.379995    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:00.383803 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.383781    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Sep 04 22:10:00.383803 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.383799    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-m6pkl\""
Sep 04 22:10:00.383964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.383832    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Sep 04 22:10:00.383964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.383836    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 04 22:10:00.384097 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.384068    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 04 22:10:00.402397 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.402368    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"]
Sep 04 22:10:00.425235 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425201    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85c7d188-f68c-4717-884d-fbf00586d73d-snapshots\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.425235 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85c7d188-f68c-4717-884d-fbf00586d73d-serving-cert\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.425539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425267    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:00.425539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425300    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tznfg\" (UniqueName: \"kubernetes.io/projected/85c7d188-f68c-4717-884d-fbf00586d73d-kube-api-access-tznfg\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.425539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425390    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:00.425539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425433    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85c7d188-f68c-4717-884d-fbf00586d73d-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.425539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425456    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85c7d188-f68c-4717-884d-fbf00586d73d-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.425539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425498    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85c7d188-f68c-4717-884d-fbf00586d73d-tmp\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.425539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425527    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxqtd\" (UniqueName: \"kubernetes.io/projected/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-kube-api-access-cxqtd\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:00.425904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/85c7d188-f68c-4717-884d-fbf00586d73d-tmp\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.425950 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.425885    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/85c7d188-f68c-4717-884d-fbf00586d73d-snapshots\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.426084 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.426067    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85c7d188-f68c-4717-884d-fbf00586d73d-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.426750 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.426729    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/85c7d188-f68c-4717-884d-fbf00586d73d-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.427810 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.427790    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/85c7d188-f68c-4717-884d-fbf00586d73d-serving-cert\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.457610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.457572    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tznfg\" (UniqueName: \"kubernetes.io/projected/85c7d188-f68c-4717-884d-fbf00586d73d-kube-api-access-tznfg\") pod \"insights-operator-76bcd6ccf5-58wlk\" (UID: \"85c7d188-f68c-4717-884d-fbf00586d73d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.526048 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.525952    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:00.526252 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.526105    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxqtd\" (UniqueName: \"kubernetes.io/projected/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-kube-api-access-cxqtd\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:00.526252 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.526155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:00.526252 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:00.526120    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:00.526402 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:00.526266    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls podName:1fbbc3a8-c52a-4592-b802-1b64b5c2a23f nodeName:}" failed. No retries permitted until 2026-09-04 22:10:01.026250921 +0000 UTC m=+84.544973431 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q4jht" (UID: "1fbbc3a8-c52a-4592-b802-1b64b5c2a23f") : secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:00.526867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.526841    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-telemetry-config\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:00.541506 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.541474    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxqtd\" (UniqueName: \"kubernetes.io/projected/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-kube-api-access-cxqtd\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:00.570339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.570310    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk"
Sep 04 22:10:00.580775 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.580748    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:00.677618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.677570    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:00.843126 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:00.843035    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-58wlk"]
Sep 04 22:10:00.846611 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:00.846584    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c7d188_f68c_4717_884d_fbf00586d73d.slice/crio-9fd140cb32ead03d076b23954508a3d86d04526804eaa548c898bd97174a4051 WatchSource:0}: Error finding container 9fd140cb32ead03d076b23954508a3d86d04526804eaa548c898bd97174a4051: Status 404 returned error can't find the container with id 9fd140cb32ead03d076b23954508a3d86d04526804eaa548c898bd97174a4051
Sep 04 22:10:01.030957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.030912    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:01.031164 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:01.031110    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:01.031211 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:01.031191    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls podName:1fbbc3a8-c52a-4592-b802-1b64b5c2a23f nodeName:}" failed. No retries permitted until 2026-09-04 22:10:02.031176041 +0000 UTC m=+85.549898552 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q4jht" (UID: "1fbbc3a8-c52a-4592-b802-1b64b5c2a23f") : secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:01.179982 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.179899    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-85b454874b-mj6ng"]
Sep 04 22:10:01.182591 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.182574    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.185141 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.185116    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Sep 04 22:10:01.185383 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.185364    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Sep 04 22:10:01.185498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.185421    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Sep 04 22:10:01.185498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.185424    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-qqqsz\""
Sep 04 22:10:01.185498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.185424    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Sep 04 22:10:01.190685 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.190667    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Sep 04 22:10:01.194476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.194455    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-mj6ng"]
Sep 04 22:10:01.232617 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.232584    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb72a802-417c-458b-bca5-c291adeb9bd1-config\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.232617 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.232618    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb72a802-417c-458b-bca5-c291adeb9bd1-serving-cert\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.232808 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.232676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr7jm\" (UniqueName: \"kubernetes.io/projected/cb72a802-417c-458b-bca5-c291adeb9bd1-kube-api-access-vr7jm\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.232808 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.232703    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb72a802-417c-458b-bca5-c291adeb9bd1-trusted-ca\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.293440 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.293399    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk" event={"ID":"85c7d188-f68c-4717-884d-fbf00586d73d","Type":"ContainerStarted","Data":"9fd140cb32ead03d076b23954508a3d86d04526804eaa548c898bd97174a4051"}
Sep 04 22:10:01.332955 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.332923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb72a802-417c-458b-bca5-c291adeb9bd1-config\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.332955 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.332958    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb72a802-417c-458b-bca5-c291adeb9bd1-serving-cert\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.333229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.333205    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr7jm\" (UniqueName: \"kubernetes.io/projected/cb72a802-417c-458b-bca5-c291adeb9bd1-kube-api-access-vr7jm\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.333283 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.333256    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb72a802-417c-458b-bca5-c291adeb9bd1-trusted-ca\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.333765 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.333744    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/cb72a802-417c-458b-bca5-c291adeb9bd1-config\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.334351 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.334335    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cb72a802-417c-458b-bca5-c291adeb9bd1-trusted-ca\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.335335 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.335318    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cb72a802-417c-458b-bca5-c291adeb9bd1-serving-cert\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.342337 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.342312    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr7jm\" (UniqueName: \"kubernetes.io/projected/cb72a802-417c-458b-bca5-c291adeb9bd1-kube-api-access-vr7jm\") pod \"console-operator-85b454874b-mj6ng\" (UID: \"cb72a802-417c-458b-bca5-c291adeb9bd1\") " pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.493187 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.493143    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:01.759125 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:01.759025    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-85b454874b-mj6ng"]
Sep 04 22:10:01.762321 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:01.762295    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb72a802_417c_458b_bca5_c291adeb9bd1.slice/crio-8b1f7286333a2e9279c10ff312166e864530c448f93519fb10c5fed40788b74d WatchSource:0}: Error finding container 8b1f7286333a2e9279c10ff312166e864530c448f93519fb10c5fed40788b74d: Status 404 returned error can't find the container with id 8b1f7286333a2e9279c10ff312166e864530c448f93519fb10c5fed40788b74d
Sep 04 22:10:02.039112 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:02.038986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:02.039272 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:02.039156    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:02.039272 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:02.039238    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls podName:1fbbc3a8-c52a-4592-b802-1b64b5c2a23f nodeName:}" failed. No retries permitted until 2026-09-04 22:10:04.039219255 +0000 UTC m=+87.557941773 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q4jht" (UID: "1fbbc3a8-c52a-4592-b802-1b64b5c2a23f") : secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:02.297640 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:02.297563    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" event={"ID":"cb72a802-417c-458b-bca5-c291adeb9bd1","Type":"ContainerStarted","Data":"8b1f7286333a2e9279c10ff312166e864530c448f93519fb10c5fed40788b74d"}
Sep 04 22:10:03.156859 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.156808    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:03.172326 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.172288    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-8d54ffd4b-bs4n8"]
Sep 04 22:10:03.174568 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.174543    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.177404 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.177173    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 04 22:10:03.177404 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.177233    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2bk2j\""
Sep 04 22:10:03.177404 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.177240    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 04 22:10:03.177945 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.177623    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 04 22:10:03.182793 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.182686    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 04 22:10:03.191807 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.191779    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-8d54ffd4b-bs4n8"]
Sep 04 22:10:03.222494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.222456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:03.248428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.248384    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-certificates\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.248428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.248421    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-trusted-ca\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.248655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.248459    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-bound-sa-token\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.248655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.248562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.248655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.248596    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-installation-pull-secrets\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.248655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.248626    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-ca-trust-extracted\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.248833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.248697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-image-registry-private-configuration\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.248833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.248728    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78zn4\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-kube-api-access-78zn4\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.302175 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.302137    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk" event={"ID":"85c7d188-f68c-4717-884d-fbf00586d73d","Type":"ContainerStarted","Data":"7f9883ceba36abc1ad986a736d3d12f7762370543684719aa6b749b7486516bd"}
Sep 04 22:10:03.349356 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.349321    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-bound-sa-token\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.349531 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.349414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.349531 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.349442    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-installation-pull-secrets\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.349531 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.349466    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-ca-trust-extracted\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.349531 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.349505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-image-registry-private-configuration\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.349531 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.349526    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78zn4\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-kube-api-access-78zn4\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.349773 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.349594    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-certificates\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.349773 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.349615    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-trusted-ca\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.350423 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:03.350108    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 22:10:03.350423 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:03.350132    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d54ffd4b-bs4n8: secret "image-registry-tls" not found
Sep 04 22:10:03.350423 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:03.350190    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls podName:bc67c976-3d0c-458a-bcc9-4f7fc06b085d nodeName:}" failed. No retries permitted until 2026-09-04 22:10:03.850170894 +0000 UTC m=+87.368893418 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls") pod "image-registry-8d54ffd4b-bs4n8" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d") : secret "image-registry-tls" not found
Sep 04 22:10:03.350670 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.350571    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-trusted-ca\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.350895 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.350861    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-ca-trust-extracted\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.351206 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.351187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-certificates\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.352923 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.352895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-image-registry-private-configuration\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.353010 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.352913    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-installation-pull-secrets\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.359874 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.359848    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78zn4\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-kube-api-access-78zn4\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.360302 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.360280    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-bound-sa-token\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.854773 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:03.854730    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:03.854955 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:03.854907    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 22:10:03.854955 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:03.854931    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d54ffd4b-bs4n8: secret "image-registry-tls" not found
Sep 04 22:10:03.855079 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:03.855002    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls podName:bc67c976-3d0c-458a-bcc9-4f7fc06b085d nodeName:}" failed. No retries permitted until 2026-09-04 22:10:04.854980078 +0000 UTC m=+88.373702595 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls") pod "image-registry-8d54ffd4b-bs4n8" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d") : secret "image-registry-tls" not found
Sep 04 22:10:04.056820 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.056772    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:04.057009 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.056937    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:04.057084 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.057022    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls podName:1fbbc3a8-c52a-4592-b802-1b64b5c2a23f nodeName:}" failed. No retries permitted until 2026-09-04 22:10:08.057002406 +0000 UTC m=+91.575724940 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q4jht" (UID: "1fbbc3a8-c52a-4592-b802-1b64b5c2a23f") : secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:04.307533 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.307444    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" event={"ID":"cb72a802-417c-458b-bca5-c291adeb9bd1","Type":"ContainerStarted","Data":"933fc8ff91ad42199f2329ec8d1ef599d53206948cb7326125416a45ae9b34e9"}
Sep 04 22:10:04.324140 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.324080    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk" podStartSLOduration=2.543142976 podStartE2EDuration="4.324039572s" totalImagesPullingTime="1.780896596s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:03.320737196 +0000 UTC m=+86.839459731" watchObservedRunningTime="2026-09-04 22:10:04.324039572 +0000 UTC m=+87.842762107"
Sep 04 22:10:04.324928 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.324897    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5bmbz"]
Sep 04 22:10:04.327529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.327491    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.328998 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.328968    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:04.331377 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.331352    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fhpsq\""
Sep 04 22:10:04.331628 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.331608    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 04 22:10:04.331841 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.331823    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 04 22:10:04.338348 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.338294    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" podStartSLOduration=0.887881499 podStartE2EDuration="3.338275838s" totalImagesPullingTime="2.450394339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:04.334311016 +0000 UTC m=+87.853033550" watchObservedRunningTime="2026-09-04 22:10:04.338275838 +0000 UTC m=+87.856998373"
Sep 04 22:10:04.338942 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.338917    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5bmbz"]
Sep 04 22:10:04.360382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.360339    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.360517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.360418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bcx8\" (UniqueName: \"kubernetes.io/projected/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-api-access-9bcx8\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.360517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.360440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4735f7ca-25fc-47b6-9f79-1479438a2a53-data-volume\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.360645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.360622    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.360771 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.360672    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4735f7ca-25fc-47b6-9f79-1479438a2a53-crio-socket\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.461334 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.461297    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.461334 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.461333    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4735f7ca-25fc-47b6-9f79-1479438a2a53-crio-socket\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.461539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.461379    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.461539 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.461421    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 22:10:04.461539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.461447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/4735f7ca-25fc-47b6-9f79-1479438a2a53-crio-socket\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.461539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.461421    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bcx8\" (UniqueName: \"kubernetes.io/projected/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-api-access-9bcx8\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.461539 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.461497    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:04.96147839 +0000 UTC m=+88.480200904 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : configmap "kube-rbac-proxy" not found
Sep 04 22:10:04.461727 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.461550    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:04.461727 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.461562    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4735f7ca-25fc-47b6-9f79-1479438a2a53-data-volume\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.461727 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.461631    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:04.961605789 +0000 UTC m=+88.480328315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:04.461911 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.461895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/4735f7ca-25fc-47b6-9f79-1479438a2a53-data-volume\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.469677 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.469596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:04.470924 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.470907    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bcx8\" (UniqueName: \"kubernetes.io/projected/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-api-access-9bcx8\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.865159 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.865073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:04.865293 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.865239    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 22:10:04.865293 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.865261    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d54ffd4b-bs4n8: secret "image-registry-tls" not found
Sep 04 22:10:04.865370 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.865321    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls podName:bc67c976-3d0c-458a-bcc9-4f7fc06b085d nodeName:}" failed. No retries permitted until 2026-09-04 22:10:06.865302606 +0000 UTC m=+90.384025116 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls") pod "image-registry-8d54ffd4b-bs4n8" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d") : secret "image-registry-tls" not found
Sep 04 22:10:04.965674 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.965613    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.965871 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:04.965722    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:04.965871 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.965777    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:04.965871 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.965799    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 22:10:04.965871 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.965843    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:05.965827796 +0000 UTC m=+89.484550307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:04.965871 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:04.965860    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:05.965854925 +0000 UTC m=+89.484577435 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : configmap "kube-rbac-proxy" not found
Sep 04 22:10:05.310925 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:05.310897    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/0.log"
Sep 04 22:10:05.311440 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:05.310938    2576 generic.go:356] "Generic (PLEG): container finished" podID="cb72a802-417c-458b-bca5-c291adeb9bd1" containerID="933fc8ff91ad42199f2329ec8d1ef599d53206948cb7326125416a45ae9b34e9" exitCode=255
Sep 04 22:10:05.311440 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:05.310971    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" event={"ID":"cb72a802-417c-458b-bca5-c291adeb9bd1","Type":"ContainerDied","Data":"933fc8ff91ad42199f2329ec8d1ef599d53206948cb7326125416a45ae9b34e9"}
Sep 04 22:10:05.311440 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:05.311333    2576 scope.go:117] "RemoveContainer" containerID="933fc8ff91ad42199f2329ec8d1ef599d53206948cb7326125416a45ae9b34e9"
Sep 04 22:10:05.973268 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:05.973224    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:05.973451 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:05.973288    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:05.973451 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:05.973363    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 22:10:05.973451 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:05.973375    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:05.973451 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:05.973424    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:07.973410131 +0000 UTC m=+91.492132662 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : configmap "kube-rbac-proxy" not found
Sep 04 22:10:05.973451 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:05.973441    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:07.973433092 +0000 UTC m=+91.492155603 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:06.071344 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:06.071318    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5mt7k_862daba0-07cf-499c-b35f-6a0d34c49cc9/dns-node-resolver/0.log"
Sep 04 22:10:06.316724 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:06.316647    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/1.log"
Sep 04 22:10:06.317146 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:06.317090    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/0.log"
Sep 04 22:10:06.317146 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:06.317123    2576 generic.go:356] "Generic (PLEG): container finished" podID="cb72a802-417c-458b-bca5-c291adeb9bd1" containerID="e91e22b6d5b90a7d0fa271dc5c58067ddccc1770fc8fb0724b10071154eb4d92" exitCode=255
Sep 04 22:10:06.317223 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:06.317165    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" event={"ID":"cb72a802-417c-458b-bca5-c291adeb9bd1","Type":"ContainerDied","Data":"e91e22b6d5b90a7d0fa271dc5c58067ddccc1770fc8fb0724b10071154eb4d92"}
Sep 04 22:10:06.317262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:06.317230    2576 scope.go:117] "RemoveContainer" containerID="933fc8ff91ad42199f2329ec8d1ef599d53206948cb7326125416a45ae9b34e9"
Sep 04 22:10:06.317428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:06.317412    2576 scope.go:117] "RemoveContainer" containerID="e91e22b6d5b90a7d0fa271dc5c58067ddccc1770fc8fb0724b10071154eb4d92"
Sep 04 22:10:06.317628 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:06.317603    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mj6ng_openshift-console-operator(cb72a802-417c-458b-bca5-c291adeb9bd1)\"" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" podUID="cb72a802-417c-458b-bca5-c291adeb9bd1"
Sep 04 22:10:06.881936 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:06.881898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:06.882134 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:06.882066    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 22:10:06.882134 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:06.882087    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d54ffd4b-bs4n8: secret "image-registry-tls" not found
Sep 04 22:10:06.882134 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:06.882135    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls podName:bc67c976-3d0c-458a-bcc9-4f7fc06b085d nodeName:}" failed. No retries permitted until 2026-09-04 22:10:10.882119222 +0000 UTC m=+94.400841738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls") pod "image-registry-8d54ffd4b-bs4n8" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d") : secret "image-registry-tls" not found
Sep 04 22:10:07.320866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:07.320842    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/1.log"
Sep 04 22:10:07.321288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:07.321184    2576 scope.go:117] "RemoveContainer" containerID="e91e22b6d5b90a7d0fa271dc5c58067ddccc1770fc8fb0724b10071154eb4d92"
Sep 04 22:10:07.321348 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:07.321330    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mj6ng_openshift-console-operator(cb72a802-417c-458b-bca5-c291adeb9bd1)\"" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" podUID="cb72a802-417c-458b-bca5-c291adeb9bd1"
Sep 04 22:10:07.471462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:07.471430    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ftrbj_8add73da-6207-468b-a38f-5db9c6261331/node-ca/0.log"
Sep 04 22:10:07.871158 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:07.871126    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-76mz2_f6434544-e7c3-4235-850c-3d4471c2c030/serve-healthcheck-canary/0.log"
Sep 04 22:10:07.990436 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:07.990394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:07.990581 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:07.990480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:07.990581 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:07.990552    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:07.990665 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:07.990613    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:11.990598792 +0000 UTC m=+95.509321310 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:07.990665 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:07.990557    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 22:10:07.990747 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:07.990695    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:11.990683638 +0000 UTC m=+95.509406149 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : configmap "kube-rbac-proxy" not found
Sep 04 22:10:08.091555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:08.091507    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:08.091739 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:08.091656    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:08.091739 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:08.091720    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls podName:1fbbc3a8-c52a-4592-b802-1b64b5c2a23f nodeName:}" failed. No retries permitted until 2026-09-04 22:10:16.091705062 +0000 UTC m=+99.610427573 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q4jht" (UID: "1fbbc3a8-c52a-4592-b802-1b64b5c2a23f") : secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:08.894885 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:08.894845    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:08.993957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:08.993919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:09.934546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:09.934514    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:10.034943 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:10.034886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:10.915370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:10.915340    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:10.915590 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:10.915469    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 22:10:10.915590 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:10.915480    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d54ffd4b-bs4n8: secret "image-registry-tls" not found
Sep 04 22:10:10.915590 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:10.915527    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls podName:bc67c976-3d0c-458a-bcc9-4f7fc06b085d nodeName:}" failed. No retries permitted until 2026-09-04 22:10:18.915514084 +0000 UTC m=+102.434236608 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls") pod "image-registry-8d54ffd4b-bs4n8" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d") : secret "image-registry-tls" not found
Sep 04 22:10:11.493425 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:11.493371    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:11.493822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:11.493754    2576 scope.go:117] "RemoveContainer" containerID="e91e22b6d5b90a7d0fa271dc5c58067ddccc1770fc8fb0724b10071154eb4d92"
Sep 04 22:10:11.493914 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:11.493895    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mj6ng_openshift-console-operator(cb72a802-417c-458b-bca5-c291adeb9bd1)\"" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" podUID="cb72a802-417c-458b-bca5-c291adeb9bd1"
Sep 04 22:10:11.991790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:11.991761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:12.023913 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:12.023874    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:12.024092 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:12.023955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:12.024148 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:12.024084    2576 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 04 22:10:12.024209 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:12.024097    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:12.024209 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:12.024185    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:20.024158857 +0000 UTC m=+103.542881392 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : configmap "kube-rbac-proxy" not found
Sep 04 22:10:12.024304 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:12.024235    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:20.024217362 +0000 UTC m=+103.542939875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:12.088526 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:12.088492    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:12.187876 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:12.187843    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:12.246084 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:12.245983    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:12.738689 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:12.738648    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:12.815236 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:12.815201    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:14.282117 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.282083    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-q57p5"]
Sep 04 22:10:14.285157 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.285139    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.287835 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.287811    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pvdlb\""
Sep 04 22:10:14.287948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.287873    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 04 22:10:14.287948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.287873    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 04 22:10:14.288864 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.288847    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 04 22:10:14.288978 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.288882    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 04 22:10:14.293641 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.293605    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-q57p5"]
Sep 04 22:10:14.307634 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.307612    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:14.307968 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.307954    2576 scope.go:117] "RemoveContainer" containerID="e91e22b6d5b90a7d0fa271dc5c58067ddccc1770fc8fb0724b10071154eb4d92"
Sep 04 22:10:14.308147 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:14.308128    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-85b454874b-mj6ng_openshift-console-operator(cb72a802-417c-458b-bca5-c291adeb9bd1)\"" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" podUID="cb72a802-417c-458b-bca5-c291adeb9bd1"
Sep 04 22:10:14.343451 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.343429    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r5d7\" (UniqueName: \"kubernetes.io/projected/183daacf-b1d0-40b4-a153-ab96c487ee00-kube-api-access-4r5d7\") pod \"service-ca-75d99c484-q57p5\" (UID: \"183daacf-b1d0-40b4-a153-ab96c487ee00\") " pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.343539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.343465    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/183daacf-b1d0-40b4-a153-ab96c487ee00-signing-cabundle\") pod \"service-ca-75d99c484-q57p5\" (UID: \"183daacf-b1d0-40b4-a153-ab96c487ee00\") " pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.343745 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.343720    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/183daacf-b1d0-40b4-a153-ab96c487ee00-signing-key\") pod \"service-ca-75d99c484-q57p5\" (UID: \"183daacf-b1d0-40b4-a153-ab96c487ee00\") " pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.383534 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.383507    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:14.444650 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.444621    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/183daacf-b1d0-40b4-a153-ab96c487ee00-signing-key\") pod \"service-ca-75d99c484-q57p5\" (UID: \"183daacf-b1d0-40b4-a153-ab96c487ee00\") " pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.444795 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.444665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r5d7\" (UniqueName: \"kubernetes.io/projected/183daacf-b1d0-40b4-a153-ab96c487ee00-kube-api-access-4r5d7\") pod \"service-ca-75d99c484-q57p5\" (UID: \"183daacf-b1d0-40b4-a153-ab96c487ee00\") " pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.444852 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.444794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/183daacf-b1d0-40b4-a153-ab96c487ee00-signing-cabundle\") pod \"service-ca-75d99c484-q57p5\" (UID: \"183daacf-b1d0-40b4-a153-ab96c487ee00\") " pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.445458 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.445436    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/183daacf-b1d0-40b4-a153-ab96c487ee00-signing-cabundle\") pod \"service-ca-75d99c484-q57p5\" (UID: \"183daacf-b1d0-40b4-a153-ab96c487ee00\") " pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.446982 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.446963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/183daacf-b1d0-40b4-a153-ab96c487ee00-signing-key\") pod \"service-ca-75d99c484-q57p5\" (UID: \"183daacf-b1d0-40b4-a153-ab96c487ee00\") " pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.455373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.455353    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r5d7\" (UniqueName: \"kubernetes.io/projected/183daacf-b1d0-40b4-a153-ab96c487ee00-kube-api-access-4r5d7\") pod \"service-ca-75d99c484-q57p5\" (UID: \"183daacf-b1d0-40b4-a153-ab96c487ee00\") " pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.484527 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.484496    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:14.594086 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.593978    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-q57p5"
Sep 04 22:10:14.747048 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.747014    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:10:14.747229 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:14.747157    2576 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 04 22:10:14.747229 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:14.747216    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls podName:2faabeaa-73ea-4a77-b707-765541109e4d nodeName:}" failed. No retries permitted until 2026-09-04 22:11:18.7472002 +0000 UTC m=+162.265922716 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls") pod "dns-default-79rjk" (UID: "2faabeaa-73ea-4a77-b707-765541109e4d") : secret "dns-default-metrics-tls" not found
Sep 04 22:10:14.860634 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:14.860567    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-q57p5"]
Sep 04 22:10:14.863748 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:14.863720    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod183daacf_b1d0_40b4_a153_ab96c487ee00.slice/crio-26929497b06b6cbe2f98c0003e9f4edffc6ec90397d2414936b1fa6d3c9511e7 WatchSource:0}: Error finding container 26929497b06b6cbe2f98c0003e9f4edffc6ec90397d2414936b1fa6d3c9511e7: Status 404 returned error can't find the container with id 26929497b06b6cbe2f98c0003e9f4edffc6ec90397d2414936b1fa6d3c9511e7
Sep 04 22:10:15.341659 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:15.341614    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-q57p5" event={"ID":"183daacf-b1d0-40b4-a153-ab96c487ee00","Type":"ContainerStarted","Data":"26929497b06b6cbe2f98c0003e9f4edffc6ec90397d2414936b1fa6d3c9511e7"}
Sep 04 22:10:15.352235 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:15.352203    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:16.157957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:16.157922    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:16.158154 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:16.158088    2576 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:16.158213 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:16.158158    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls podName:1fbbc3a8-c52a-4592-b802-1b64b5c2a23f nodeName:}" failed. No retries permitted until 2026-09-04 22:10:32.15814137 +0000 UTC m=+115.676863880 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-6cdcc96965-q4jht" (UID: "1fbbc3a8-c52a-4592-b802-1b64b5c2a23f") : secret "cluster-monitoring-operator-tls" not found
Sep 04 22:10:17.350346 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:17.350307    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-q57p5" event={"ID":"183daacf-b1d0-40b4-a153-ab96c487ee00","Type":"ContainerStarted","Data":"40fdd914f10819eb786692c3af3aaca38c7d044fe75714a14bbd370acb435fe1"}
Sep 04 22:10:17.370889 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:17.370831    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-q57p5" podStartSLOduration=1.483625205 podStartE2EDuration="3.370811997s" totalImagesPullingTime="1.887186792s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:17.369449549 +0000 UTC m=+100.888172085" watchObservedRunningTime="2026-09-04 22:10:17.370811997 +0000 UTC m=+100.889534531"
Sep 04 22:10:17.521950 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:17.521913    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:18.456610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:18.456573    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:18.917967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:18.917895    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:18.979496 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:18.979460    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:18.980261 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:18.979780    2576 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 04 22:10:18.980421 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:18.980410    2576 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-8d54ffd4b-bs4n8: secret "image-registry-tls" not found
Sep 04 22:10:18.980552 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:18.980543    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls podName:bc67c976-3d0c-458a-bcc9-4f7fc06b085d nodeName:}" failed. No retries permitted until 2026-09-04 22:10:34.980523073 +0000 UTC m=+118.499245591 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls") pod "image-registry-8d54ffd4b-bs4n8" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d") : secret "image-registry-tls" not found
Sep 04 22:10:19.102587 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:19.102549    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:19.172796 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:19.172709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:19.252750 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:19.252714    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:19.265480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:19.265460    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kr6k4"
Sep 04 22:10:19.451537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:19.451425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:19.627045 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:19.626962    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:19.722675 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:19.722637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:20.024754 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:20.024653    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:20.089629 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:20.089586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:20.089781 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:20.089658    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:20.089844 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:20.089809    2576 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:20.089890 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:20.089879    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls podName:4735f7ca-25fc-47b6-9f79-1479438a2a53 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:36.089860756 +0000 UTC m=+119.608583282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5bmbz" (UID: "4735f7ca-25fc-47b6-9f79-1479438a2a53") : secret "insights-runtime-extractor-tls" not found
Sep 04 22:10:20.090127 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:20.090111    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/4735f7ca-25fc-47b6-9f79-1479438a2a53-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:21.016464 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:21.016427    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:21.416099 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:21.415975    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:21.621539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:21.621506    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:21.816386 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:21.816337    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:22.216584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:22.216552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:22.616114 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:22.616002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:22.723911 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:22.723849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:23.012034 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:23.012004    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:23.202540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:23.202505    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:23.260918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:23.260888    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:23.415728 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:23.415637    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:23.615992 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:23.615956    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:23.675816 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:23.675740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:23.817306 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:23.817270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:24.023603 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:24.023562    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:25.019142 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:25.019106    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:25.240106 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:25.240050    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:25.422479 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:25.422385    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:25.609760 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:25.609722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:25.812820 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:25.812788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:26.212115 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:26.212082    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:26.814875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:26.814837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:27.226068 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:27.226034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:27.425653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:27.425614    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:28.023071 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:28.023024    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:28.102918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:28.102884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:28.614415 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:28.614379    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:28.835255 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:28.835225    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:28.912935 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:28.912870    2576 scope.go:117] "RemoveContainer" containerID="e91e22b6d5b90a7d0fa271dc5c58067ddccc1770fc8fb0724b10071154eb4d92"
Sep 04 22:10:29.016716 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:29.016690    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:29.387445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:29.387417    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:10:29.387817 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:29.387801    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/1.log"
Sep 04 22:10:29.387867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:29.387834    2576 generic.go:356] "Generic (PLEG): container finished" podID="cb72a802-417c-458b-bca5-c291adeb9bd1" containerID="e3a7a9f1c5e937bedf694adf37df366b7d50ff7b78c4c43a3c73c7b63425968a" exitCode=255
Sep 04 22:10:29.387906 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:29.387866    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" event={"ID":"cb72a802-417c-458b-bca5-c291adeb9bd1","Type":"ContainerDied","Data":"e3a7a9f1c5e937bedf694adf37df366b7d50ff7b78c4c43a3c73c7b63425968a"}
Sep 04 22:10:29.387939 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:29.387907    2576 scope.go:117] "RemoveContainer" containerID="e91e22b6d5b90a7d0fa271dc5c58067ddccc1770fc8fb0724b10071154eb4d92"
Sep 04 22:10:29.388227 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:29.388208    2576 scope.go:117] "RemoveContainer" containerID="e3a7a9f1c5e937bedf694adf37df366b7d50ff7b78c4c43a3c73c7b63425968a"
Sep 04 22:10:29.388417 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:29.388392    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-mj6ng_openshift-console-operator(cb72a802-417c-458b-bca5-c291adeb9bd1)\"" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" podUID="cb72a802-417c-458b-bca5-c291adeb9bd1"
Sep 04 22:10:29.413174 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:29.413147    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:29.616541 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:29.616505    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:30.392019 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:30.391988    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:10:31.493614 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:31.493572    2576 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:31.494074 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:31.493905    2576 scope.go:117] "RemoveContainer" containerID="e3a7a9f1c5e937bedf694adf37df366b7d50ff7b78c4c43a3c73c7b63425968a"
Sep 04 22:10:31.494122 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:31.494050    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-mj6ng_openshift-console-operator(cb72a802-417c-458b-bca5-c291adeb9bd1)\"" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" podUID="cb72a802-417c-458b-bca5-c291adeb9bd1"
Sep 04 22:10:32.188009 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:32.187951    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:32.190348 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:32.190326    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1fbbc3a8-c52a-4592-b802-1b64b5c2a23f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-6cdcc96965-q4jht\" (UID: \"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f\") " pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:32.488258 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:32.488223    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"
Sep 04 22:10:32.755779 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:32.755699    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht"]
Sep 04 22:10:32.759528 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:32.759506    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fbbc3a8_c52a_4592_b802_1b64b5c2a23f.slice/crio-c1f0e6968fec9f0408094b122145b7eb304e4572ca471707470504c3dd22985a WatchSource:0}: Error finding container c1f0e6968fec9f0408094b122145b7eb304e4572ca471707470504c3dd22985a: Status 404 returned error can't find the container with id c1f0e6968fec9f0408094b122145b7eb304e4572ca471707470504c3dd22985a
Sep 04 22:10:33.401538 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:33.401506    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht" event={"ID":"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f","Type":"ContainerStarted","Data":"c1f0e6968fec9f0408094b122145b7eb304e4572ca471707470504c3dd22985a"}
Sep 04 22:10:34.307772 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:34.307515    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:34.308200 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:34.307949    2576 scope.go:117] "RemoveContainer" containerID="e3a7a9f1c5e937bedf694adf37df366b7d50ff7b78c4c43a3c73c7b63425968a"
Sep 04 22:10:34.308200 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:34.308120    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 20s restarting failed container=console-operator pod=console-operator-85b454874b-mj6ng_openshift-console-operator(cb72a802-417c-458b-bca5-c291adeb9bd1)\"" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" podUID="cb72a802-417c-458b-bca5-c291adeb9bd1"
Sep 04 22:10:34.838657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:34.838616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:34.938304 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:34.938267    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:35.013444 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.013398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:35.015921 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.015899    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"image-registry-8d54ffd4b-bs4n8\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") " pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:35.039828 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.039788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:35.077564 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.077527    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qt6pg"]
Sep 04 22:10:35.079452 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.079434    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg"
Sep 04 22:10:35.081148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.081123    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"]
Sep 04 22:10:35.082585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.082563    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 04 22:10:35.082809 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.082790    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 04 22:10:35.083368 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.083352    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.083909 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.083888    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-6hbkg\""
Sep 04 22:10:35.086087 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.086045    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 04 22:10:35.086434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.086416    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 04 22:10:35.086861 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.086686    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 04 22:10:35.087237 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.087219    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 04 22:10:35.098475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.098396    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qt6pg"]
Sep 04 22:10:35.099609 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.099584    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"]
Sep 04 22:10:35.109254 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.109230    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8d54ffd4b-bs4n8"]
Sep 04 22:10:35.109456 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.109442    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:35.178946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.178699    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"]
Sep 04 22:10:35.180932 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.180914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"
Sep 04 22:10:35.187315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.187295    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Sep 04 22:10:35.187422 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.187315    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-ttxv2\""
Sep 04 22:10:35.196619 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.196597    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"]
Sep 04 22:10:35.215797 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.215768    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e302d63-80ad-4370-8feb-17b2bf538ee2-nginx-conf\") pod \"networking-console-plugin-79977f9844-qt6pg\" (UID: \"4e302d63-80ad-4370-8feb-17b2bf538ee2\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg"
Sep 04 22:10:35.215908 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.215819    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4e302d63-80ad-4370-8feb-17b2bf538ee2-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qt6pg\" (UID: \"4e302d63-80ad-4370-8feb-17b2bf538ee2\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg"
Sep 04 22:10:35.215968 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.215944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a7eec48-a7c6-43ec-875d-7f9b7412d091-tmp\") pod \"klusterlet-addon-workmgr-67dd46f9c7-hxwcf\" (UID: \"2a7eec48-a7c6-43ec-875d-7f9b7412d091\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.216027 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.215969    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2a7eec48-a7c6-43ec-875d-7f9b7412d091-klusterlet-config\") pod \"klusterlet-addon-workmgr-67dd46f9c7-hxwcf\" (UID: \"2a7eec48-a7c6-43ec-875d-7f9b7412d091\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.216027 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.215989    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rwrc\" (UniqueName: \"kubernetes.io/projected/2a7eec48-a7c6-43ec-875d-7f9b7412d091-kube-api-access-6rwrc\") pod \"klusterlet-addon-workmgr-67dd46f9c7-hxwcf\" (UID: \"2a7eec48-a7c6-43ec-875d-7f9b7412d091\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.236355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.236329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:35.317316 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.317280    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c996da1e-c340-4079-bec0-af1f00551466-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-962mh\" (UID: \"c996da1e-c340-4079-bec0-af1f00551466\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"
Sep 04 22:10:35.317778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.317352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a7eec48-a7c6-43ec-875d-7f9b7412d091-tmp\") pod \"klusterlet-addon-workmgr-67dd46f9c7-hxwcf\" (UID: \"2a7eec48-a7c6-43ec-875d-7f9b7412d091\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.317778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.317378    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2a7eec48-a7c6-43ec-875d-7f9b7412d091-klusterlet-config\") pod \"klusterlet-addon-workmgr-67dd46f9c7-hxwcf\" (UID: \"2a7eec48-a7c6-43ec-875d-7f9b7412d091\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.317778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.317405    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rwrc\" (UniqueName: \"kubernetes.io/projected/2a7eec48-a7c6-43ec-875d-7f9b7412d091-kube-api-access-6rwrc\") pod \"klusterlet-addon-workmgr-67dd46f9c7-hxwcf\" (UID: \"2a7eec48-a7c6-43ec-875d-7f9b7412d091\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.317778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.317453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e302d63-80ad-4370-8feb-17b2bf538ee2-nginx-conf\") pod \"networking-console-plugin-79977f9844-qt6pg\" (UID: \"4e302d63-80ad-4370-8feb-17b2bf538ee2\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg"
Sep 04 22:10:35.317778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.317488    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4e302d63-80ad-4370-8feb-17b2bf538ee2-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qt6pg\" (UID: \"4e302d63-80ad-4370-8feb-17b2bf538ee2\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg"
Sep 04 22:10:35.318023 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.317817    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a7eec48-a7c6-43ec-875d-7f9b7412d091-tmp\") pod \"klusterlet-addon-workmgr-67dd46f9c7-hxwcf\" (UID: \"2a7eec48-a7c6-43ec-875d-7f9b7412d091\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.318239 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.318216    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e302d63-80ad-4370-8feb-17b2bf538ee2-nginx-conf\") pod \"networking-console-plugin-79977f9844-qt6pg\" (UID: \"4e302d63-80ad-4370-8feb-17b2bf538ee2\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg"
Sep 04 22:10:35.319967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.319946    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4e302d63-80ad-4370-8feb-17b2bf538ee2-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-qt6pg\" (UID: \"4e302d63-80ad-4370-8feb-17b2bf538ee2\") " pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg"
Sep 04 22:10:35.320082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.319977    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2a7eec48-a7c6-43ec-875d-7f9b7412d091-klusterlet-config\") pod \"klusterlet-addon-workmgr-67dd46f9c7-hxwcf\" (UID: \"2a7eec48-a7c6-43ec-875d-7f9b7412d091\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.326231 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.326211    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:35.329317 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.329300    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rwrc\" (UniqueName: \"kubernetes.io/projected/2a7eec48-a7c6-43ec-875d-7f9b7412d091-kube-api-access-6rwrc\") pod \"klusterlet-addon-workmgr-67dd46f9c7-hxwcf\" (UID: \"2a7eec48-a7c6-43ec-875d-7f9b7412d091\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.393526 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.393458    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg"
Sep 04 22:10:35.400821 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.400785    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:35.402721 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.402565    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8d54ffd4b-bs4n8"]
Sep 04 22:10:35.404696 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:35.404675    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc67c976_3d0c_458a_bcc9_4f7fc06b085d.slice/crio-fb737e45e642eb8a2cb19b7f838ac11b5bbd9af08c9255abb5ea901251e839d7 WatchSource:0}: Error finding container fb737e45e642eb8a2cb19b7f838ac11b5bbd9af08c9255abb5ea901251e839d7: Status 404 returned error can't find the container with id fb737e45e642eb8a2cb19b7f838ac11b5bbd9af08c9255abb5ea901251e839d7
Sep 04 22:10:35.414153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.414118    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht" event={"ID":"1fbbc3a8-c52a-4592-b802-1b64b5c2a23f","Type":"ContainerStarted","Data":"44c732d8eba6b86ae5cbef6b20ac223ef539a9c2214b47d54ef154705d3b8527"}
Sep 04 22:10:35.418643 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.418614    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c996da1e-c340-4079-bec0-af1f00551466-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-962mh\" (UID: \"c996da1e-c340-4079-bec0-af1f00551466\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"
Sep 04 22:10:35.418809 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:35.418789    2576 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Sep 04 22:10:35.418891 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:35.418870    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c996da1e-c340-4079-bec0-af1f00551466-tls-certificates podName:c996da1e-c340-4079-bec0-af1f00551466 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:35.918849562 +0000 UTC m=+119.437572088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/c996da1e-c340-4079-bec0-af1f00551466-tls-certificates") pod "prometheus-operator-admission-webhook-78f599db79-962mh" (UID: "c996da1e-c340-4079-bec0-af1f00551466") : secret "prometheus-operator-admission-webhook-tls" not found
Sep 04 22:10:35.436616 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.436566    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-6cdcc96965-q4jht" podStartSLOduration=33.755123246 podStartE2EDuration="35.436550734s" totalImagesPullingTime="1.681427488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:35.43612522 +0000 UTC m=+118.954847753" watchObservedRunningTime="2026-09-04 22:10:35.436550734 +0000 UTC m=+118.955273266"
Sep 04 22:10:35.627065 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.627017    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:35.679857 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.679776    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-qt6pg"]
Sep 04 22:10:35.682961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:35.682934    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e302d63_80ad_4370_8feb_17b2bf538ee2.slice/crio-4776373cbf8c66c6beb1f3847f770a976a07dce36b9ed54ca8cd01ec8986c0bd WatchSource:0}: Error finding container 4776373cbf8c66c6beb1f3847f770a976a07dce36b9ed54ca8cd01ec8986c0bd: Status 404 returned error can't find the container with id 4776373cbf8c66c6beb1f3847f770a976a07dce36b9ed54ca8cd01ec8986c0bd
Sep 04 22:10:35.682961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.682942    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"]
Sep 04 22:10:35.686740 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:35.686685    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a7eec48_a7c6_43ec_875d_7f9b7412d091.slice/crio-95affb9593ae3f61c8f6c1fd5bfe8240d94f629a2c416f7cb25b18726e965fdd WatchSource:0}: Error finding container 95affb9593ae3f61c8f6c1fd5bfe8240d94f629a2c416f7cb25b18726e965fdd: Status 404 returned error can't find the container with id 95affb9593ae3f61c8f6c1fd5bfe8240d94f629a2c416f7cb25b18726e965fdd
Sep 04 22:10:35.922765 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.922731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c996da1e-c340-4079-bec0-af1f00551466-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-962mh\" (UID: \"c996da1e-c340-4079-bec0-af1f00551466\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"
Sep 04 22:10:35.925150 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:35.925131    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/c996da1e-c340-4079-bec0-af1f00551466-tls-certificates\") pod \"prometheus-operator-admission-webhook-78f599db79-962mh\" (UID: \"c996da1e-c340-4079-bec0-af1f00551466\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"
Sep 04 22:10:36.089783 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.089739    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"
Sep 04 22:10:36.124626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.124595    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:36.127518 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.127484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/4735f7ca-25fc-47b6-9f79-1479438a2a53-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5bmbz\" (UID: \"4735f7ca-25fc-47b6-9f79-1479438a2a53\") " pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:36.147714 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.147611    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5bmbz"
Sep 04 22:10:36.358167 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.358089    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"]
Sep 04 22:10:36.360779 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:36.360755    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc996da1e_c340_4079_bec0_af1f00551466.slice/crio-411be754f924c7d88f59d40878ab0ed5d08ea790ac9826aa5a05d339df9280fe WatchSource:0}: Error finding container 411be754f924c7d88f59d40878ab0ed5d08ea790ac9826aa5a05d339df9280fe: Status 404 returned error can't find the container with id 411be754f924c7d88f59d40878ab0ed5d08ea790ac9826aa5a05d339df9280fe
Sep 04 22:10:36.420559 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.420521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh" event={"ID":"c996da1e-c340-4079-bec0-af1f00551466","Type":"ContainerStarted","Data":"411be754f924c7d88f59d40878ab0ed5d08ea790ac9826aa5a05d339df9280fe"}
Sep 04 22:10:36.422072 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.422020    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg" event={"ID":"4e302d63-80ad-4370-8feb-17b2bf538ee2","Type":"ContainerStarted","Data":"4776373cbf8c66c6beb1f3847f770a976a07dce36b9ed54ca8cd01ec8986c0bd"}
Sep 04 22:10:36.424389 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.424295    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8" event={"ID":"bc67c976-3d0c-458a-bcc9-4f7fc06b085d","Type":"ContainerStarted","Data":"d47ff6cea0edc1f83974f49134dbadefb01da1344bc800ff7eb84978f3120126"}
Sep 04 22:10:36.424389 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.424341    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8" event={"ID":"bc67c976-3d0c-458a-bcc9-4f7fc06b085d","Type":"ContainerStarted","Data":"fb737e45e642eb8a2cb19b7f838ac11b5bbd9af08c9255abb5ea901251e839d7"}
Sep 04 22:10:36.424809 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.424770    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:36.428040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.428010    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5bmbz"]
Sep 04 22:10:36.431334 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.431278    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf" event={"ID":"2a7eec48-a7c6-43ec-875d-7f9b7412d091","Type":"ContainerStarted","Data":"95affb9593ae3f61c8f6c1fd5bfe8240d94f629a2c416f7cb25b18726e965fdd"}
Sep 04 22:10:36.434123 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:36.434096    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4735f7ca_25fc_47b6_9f79_1479438a2a53.slice/crio-ecb576979c28f00659e472e760b398447bcffd515cc7568726bb416e9fd2db32 WatchSource:0}: Error finding container ecb576979c28f00659e472e760b398447bcffd515cc7568726bb416e9fd2db32: Status 404 returned error can't find the container with id ecb576979c28f00659e472e760b398447bcffd515cc7568726bb416e9fd2db32
Sep 04 22:10:36.451719 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:36.451670    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8" podStartSLOduration=33.451652052 podStartE2EDuration="33.451652052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:36.451200398 +0000 UTC m=+119.969922932" watchObservedRunningTime="2026-09-04 22:10:36.451652052 +0000 UTC m=+119.970374586"
Sep 04 22:10:37.436725 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:37.436685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bmbz" event={"ID":"4735f7ca-25fc-47b6-9f79-1479438a2a53","Type":"ContainerStarted","Data":"156985fd76b97480fe3f145cd0cb6c6d9d9d740a287eb9820303d4f9fcc7ea12"}
Sep 04 22:10:37.437171 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:37.436758    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bmbz" event={"ID":"4735f7ca-25fc-47b6-9f79-1479438a2a53","Type":"ContainerStarted","Data":"ecb576979c28f00659e472e760b398447bcffd515cc7568726bb416e9fd2db32"}
Sep 04 22:10:38.207134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:38.207087    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:38.267807 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:38.267740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:40.450221 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.450184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh" event={"ID":"c996da1e-c340-4079-bec0-af1f00551466","Type":"ContainerStarted","Data":"a87118edd7bcbfaafcb39d2ff658aea0c81afd76cca1e6d379721b2a33a917c3"}
Sep 04 22:10:40.451020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.451000    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"
Sep 04 22:10:40.452701 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.452660    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg" event={"ID":"4e302d63-80ad-4370-8feb-17b2bf538ee2","Type":"ContainerStarted","Data":"901440ca0efe87eb1c47dc12dfeb6d362b5034add679b6960725c975f01baeb9"}
Sep 04 22:10:40.454750 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.454713    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf" event={"ID":"2a7eec48-a7c6-43ec-875d-7f9b7412d091","Type":"ContainerStarted","Data":"3cb71cd917db845d666f056c7d3fe4a406bb74fcfa0c02583bb375f4be000e8b"}
Sep 04 22:10:40.454984 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.454968    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:40.456611 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.456590    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf"
Sep 04 22:10:40.457116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.457095    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bmbz" event={"ID":"4735f7ca-25fc-47b6-9f79-1479438a2a53","Type":"ContainerStarted","Data":"47efdb3cecb11eb1c8e435de99b508e51f54f7d759ae53931fb613bc6a57eb3e"}
Sep 04 22:10:40.457355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.457292    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh"
Sep 04 22:10:40.470013 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.469968    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-78f599db79-962mh" podStartSLOduration=2.118485205 podStartE2EDuration="5.469956189s" totalImagesPullingTime="3.351470984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:40.469434392 +0000 UTC m=+123.988156926" watchObservedRunningTime="2026-09-04 22:10:40.469956189 +0000 UTC m=+123.988678722"
Sep 04 22:10:40.488427 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.488381    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-qt6pg" podStartSLOduration=1.464078263 podStartE2EDuration="5.488364427s" totalImagesPullingTime="4.024286164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:40.487020608 +0000 UTC m=+124.005743143" watchObservedRunningTime="2026-09-04 22:10:40.488364427 +0000 UTC m=+124.007086980"
Sep 04 22:10:40.505439 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.505386    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-67dd46f9c7-hxwcf" podStartSLOduration=1.460952479 podStartE2EDuration="5.5053685s" totalImagesPullingTime="4.044416021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:40.504226867 +0000 UTC m=+124.022949400" watchObservedRunningTime="2026-09-04 22:10:40.5053685 +0000 UTC m=+124.024091030"
Sep 04 22:10:40.609501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.609456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:40.683844 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.683801    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:40.977162 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:40.977125    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:41.079114 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:41.079081    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:41.288964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:41.288877    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:41.461879 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:41.461844    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5bmbz" event={"ID":"4735f7ca-25fc-47b6-9f79-1479438a2a53","Type":"ContainerStarted","Data":"c538d84ebc847ebeb57f6024d4a75b78ac0a46e9c91df7df71d24788cf7de9fc"}
Sep 04 22:10:41.482157 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:41.482111    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5bmbz" podStartSLOduration=33.112342345 podStartE2EDuration="37.482095523s" totalImagesPullingTime="4.369753178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:10:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:41.480840129 +0000 UTC m=+124.999562653" watchObservedRunningTime="2026-09-04 22:10:41.482095523 +0000 UTC m=+125.000818055"
Sep 04 22:10:43.317979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:43.317946    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:46.125562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.125528    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:46.222484 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.222443    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:46.366715 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.366677    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:46.394478 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.394412    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"]
Sep 04 22:10:46.398289 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.398269    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.401877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.401856    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 04 22:10:46.402087 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.402067    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-tpwnc\""
Sep 04 22:10:46.402156 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.402098    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 04 22:10:46.403325 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.403305    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 04 22:10:46.413022 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.413001    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"]
Sep 04 22:10:46.441011 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.440985    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-lwnlx"]
Sep 04 22:10:46.444577 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.444558    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.447500 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.447470    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 04 22:10:46.447597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.447476    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 04 22:10:46.448233 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.448216    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 04 22:10:46.448324 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.448284    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-fxd5g\""
Sep 04 22:10:46.476101 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.476075    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:46.511804 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.511772    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5mng\" (UniqueName: \"kubernetes.io/projected/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-kube-api-access-g5mng\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.511954 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.511837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.511954 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.511863    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.511954 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.511881    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.569685 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.569658    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:46.612705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.612673    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.612850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.612711    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.612850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.612750    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-tls\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.612850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.612782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.612850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.612811    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-sys\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.612850 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:46.612826    2576 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 04 22:10:46.613034 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.612965    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-wtmp\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.613034 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:10:46.612996    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-openshift-state-metrics-tls podName:7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1 nodeName:}" failed. No retries permitted until 2026-09-04 22:10:47.112969315 +0000 UTC m=+130.631691829 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-openshift-state-metrics-tls") pod "openshift-state-metrics-657fd6c54-5jsh6" (UID: "7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1") : secret "openshift-state-metrics-tls" not found
Sep 04 22:10:46.613156 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.613078    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.613156 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.613122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-textfile\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.613230 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.613170    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5mng\" (UniqueName: \"kubernetes.io/projected/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-kube-api-access-g5mng\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.613230 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.613184    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-root\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.613230 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.613203    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvcm\" (UniqueName: \"kubernetes.io/projected/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-kube-api-access-ltvcm\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.613341 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.613292    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.613341 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.613324    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-metrics-client-ca\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.613870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.613852    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.615154 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.615132    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.622505 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.622485    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5mng\" (UniqueName: \"kubernetes.io/projected/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-kube-api-access-g5mng\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:46.674641 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.674565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:46.713734 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-wtmp\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.713853 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713743    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.713853 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:10:46.713853 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-textfile\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.713853 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713840    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-root\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvcm\" (UniqueName: \"kubernetes.io/projected/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-kube-api-access-ltvcm\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713890    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-wtmp\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713919    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-metrics-client-ca\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713929    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-root\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.713984    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-tls\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.714021    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.714073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-sys\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.714154    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-sys\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.714188    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-textfile\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714656 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.714364    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-accelerators-collector-config\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.714656 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.714479    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-metrics-client-ca\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.716187 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.716159    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ca6195d-72ed-44c3-bc9e-449566276bfe-metrics-certs\") pod \"network-metrics-daemon-ssvlz\" (UID: \"2ca6195d-72ed-44c3-bc9e-449566276bfe\") " pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:10:46.716352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.716334    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.716627 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.716611    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-node-exporter-tls\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.732655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.732629    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvcm\" (UniqueName: \"kubernetes.io/projected/b1f78268-1b3c-4799-9e9c-c30b0a297b6c-kube-api-access-ltvcm\") pod \"node-exporter-lwnlx\" (UID: \"b1f78268-1b3c-4799-9e9c-c30b0a297b6c\") " pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.755502 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.755479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-lwnlx"
Sep 04 22:10:46.766683 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:46.766662    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f78268_1b3c_4799_9e9c_c30b0a297b6c.slice/crio-e27b882a68f73159f8cfdeb59abe34bb90144e0daf830b4aa1acf035cd6e8b1f WatchSource:0}: Error finding container e27b882a68f73159f8cfdeb59abe34bb90144e0daf830b4aa1acf035cd6e8b1f: Status 404 returned error can't find the container with id e27b882a68f73159f8cfdeb59abe34bb90144e0daf830b4aa1acf035cd6e8b1f
Sep 04 22:10:46.768976 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.768952    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:46.879217 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.879188    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:46.926152 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.926079    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-c64pf\""
Sep 04 22:10:46.934219 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.934191    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ssvlz"
Sep 04 22:10:46.994900 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:46.994860    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:47.116384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.116349    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:47.118703 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.118677    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-5jsh6\" (UID: \"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:47.211799 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.211656    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ssvlz"]
Sep 04 22:10:47.214203 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:47.214180    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca6195d_72ed_44c3_bc9e_449566276bfe.slice/crio-5b4de8ba83ff6af40e5f93b63c0c8442fe1eacc5a9e055c250cb50e77bdf6e14 WatchSource:0}: Error finding container 5b4de8ba83ff6af40e5f93b63c0c8442fe1eacc5a9e055c250cb50e77bdf6e14: Status 404 returned error can't find the container with id 5b4de8ba83ff6af40e5f93b63c0c8442fe1eacc5a9e055c250cb50e77bdf6e14
Sep 04 22:10:47.308364 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.308328    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"
Sep 04 22:10:47.326578 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.326547    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:47.426040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.426006    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:47.483097 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.482946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ssvlz" event={"ID":"2ca6195d-72ed-44c3-bc9e-449566276bfe","Type":"ContainerStarted","Data":"5b4de8ba83ff6af40e5f93b63c0c8442fe1eacc5a9e055c250cb50e77bdf6e14"}
Sep 04 22:10:47.484378 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.484349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwnlx" event={"ID":"b1f78268-1b3c-4799-9e9c-c30b0a297b6c","Type":"ContainerStarted","Data":"e27b882a68f73159f8cfdeb59abe34bb90144e0daf830b4aa1acf035cd6e8b1f"}
Sep 04 22:10:47.536770 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.536735    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 04 22:10:47.541073 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.541033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.543400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.543357    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:47.543950 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.543927    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 04 22:10:47.543950 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.543939    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 04 22:10:47.544462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.544246    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 04 22:10:47.544462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.544279    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 04 22:10:47.544462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.544314    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 04 22:10:47.544462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.544345    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 04 22:10:47.544749 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.544734    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wls69\""
Sep 04 22:10:47.544816 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.544799    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 04 22:10:47.545047 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.545029    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 04 22:10:47.545159 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.545146    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 04 22:10:47.557589 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.557567    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 04 22:10:47.597927 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.597836    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6"]
Sep 04 22:10:47.620891 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.620812    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:47.621353 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621323    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621383    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621448    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621632 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621477    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-web-config\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621632 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621510    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-out\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621632 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621568    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621782 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621625    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621782 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621654    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk7lt\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-kube-api-access-rk7lt\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621782 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621782 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621713    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.621782 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621756    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.622029 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.621798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722238 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:47.722240    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a9743d5_dbeb_4b09_ac3a_a83bad6b18c1.slice/crio-136e22fe71c213b8eab2f32026d25fe63860c433d84dd099365778d2d762909c WatchSource:0}: Error finding container 136e22fe71c213b8eab2f32026d25fe63860c433d84dd099365778d2d762909c: Status 404 returned error can't find the container with id 136e22fe71c213b8eab2f32026d25fe63860c433d84dd099365778d2d762909c
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-web-config\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722350    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-out\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722401    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722423    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk7lt\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-kube-api-access-rk7lt\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722455    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722482    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.722529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.723121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722553    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.723121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.722616    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.723537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.723475    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.725124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.723990    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.725124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.724383    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.726916 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.726778    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-web-config\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.726916 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.726879    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.727224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.727192    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-tls-assets\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.727903 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.727333    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.728364 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.728334    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.728479 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.728427    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.728577 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.728557    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-out\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.728646 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.728625    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.729255 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.729232    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-volume\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.740222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.740202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk7lt\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-kube-api-access-rk7lt\") pod \"alertmanager-main-0\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:47.852720 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:47.852524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:10:48.153218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.153046    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 04 22:10:48.157496 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:48.157467    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c0849a_f082_4085_9e32_42f6f5ea75e6.slice/crio-2b8d88f03904d85fbf7eac02a22f19b30d9de494cf86b0eab6b62312aab77438 WatchSource:0}: Error finding container 2b8d88f03904d85fbf7eac02a22f19b30d9de494cf86b0eab6b62312aab77438: Status 404 returned error can't find the container with id 2b8d88f03904d85fbf7eac02a22f19b30d9de494cf86b0eab6b62312aab77438
Sep 04 22:10:48.489407 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.489370    2576 generic.go:356] "Generic (PLEG): container finished" podID="b1f78268-1b3c-4799-9e9c-c30b0a297b6c" containerID="9aa5da47b2946a625322d73b04e6134cbf6a3b0f4b47000a4bcda5c049be35eb" exitCode=0
Sep 04 22:10:48.489780 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.489451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwnlx" event={"ID":"b1f78268-1b3c-4799-9e9c-c30b0a297b6c","Type":"ContainerDied","Data":"9aa5da47b2946a625322d73b04e6134cbf6a3b0f4b47000a4bcda5c049be35eb"}
Sep 04 22:10:48.490743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.490708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerStarted","Data":"2b8d88f03904d85fbf7eac02a22f19b30d9de494cf86b0eab6b62312aab77438"}
Sep 04 22:10:48.493685 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.493431    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6" event={"ID":"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1","Type":"ContainerStarted","Data":"333a3b3b365e426c8dc6c0d4c85f6be12fb92097bf7d23879b38654f5acbb09d"}
Sep 04 22:10:48.493685 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.493491    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6" event={"ID":"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1","Type":"ContainerStarted","Data":"e7d038bea0b8e7bf80b4c12fdb8717bea6ffb3d9a1ce0170622a70da062d8bf3"}
Sep 04 22:10:48.493685 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.493522    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6" event={"ID":"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1","Type":"ContainerStarted","Data":"136e22fe71c213b8eab2f32026d25fe63860c433d84dd099365778d2d762909c"}
Sep 04 22:10:48.509087 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.505546    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-57498c5b8c-4j56l"]
Sep 04 22:10:48.513158 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.513141    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.515936 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.515913    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Sep 04 22:10:48.515936 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.515926    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-t5minkvq3tld\""
Sep 04 22:10:48.516099 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.516023    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Sep 04 22:10:48.516204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.516115    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Sep 04 22:10:48.516204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.516136    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Sep 04 22:10:48.516204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.516145    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Sep 04 22:10:48.516349 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.516236    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-6kplz\""
Sep 04 22:10:48.524308 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.524258    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57498c5b8c-4j56l"]
Sep 04 22:10:48.528922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.528573    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.528922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.528617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.528922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.528647    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxxhw\" (UniqueName: \"kubernetes.io/projected/d92d9363-9172-4535-8eba-c8402302d2b2-kube-api-access-sxxhw\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.528922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.528735    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.528922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.528782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.529315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.528985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d92d9363-9172-4535-8eba-c8402302d2b2-metrics-client-ca\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.529315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.529086    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-grpc-tls\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.529315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.529114    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-tls\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.630798 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.630767    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d92d9363-9172-4535-8eba-c8402302d2b2-metrics-client-ca\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.630898 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.630846    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-grpc-tls\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.630898 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.630873    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-tls\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.631010 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.630928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.631010 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.630963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.631010 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.630990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxxhw\" (UniqueName: \"kubernetes.io/projected/d92d9363-9172-4535-8eba-c8402302d2b2-kube-api-access-sxxhw\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.631177 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.631041    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.631177 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.631086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.633922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.633070    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d92d9363-9172-4535-8eba-c8402302d2b2-metrics-client-ca\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.634998 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.634971    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.638753 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.638720    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.639738 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.639381    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-grpc-tls\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.640912 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.640873    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.642014 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.641560    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.642014 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.641985    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/d92d9363-9172-4535-8eba-c8402302d2b2-secret-thanos-querier-tls\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.644989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.644941    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxxhw\" (UniqueName: \"kubernetes.io/projected/d92d9363-9172-4535-8eba-c8402302d2b2-kube-api-access-sxxhw\") pod \"thanos-querier-57498c5b8c-4j56l\" (UID: \"d92d9363-9172-4535-8eba-c8402302d2b2\") " pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:48.826946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:48.826908    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:49.119525 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.119499    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-57498c5b8c-4j56l"]
Sep 04 22:10:49.127413 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:49.127386    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92d9363_9172_4535_8eba_c8402302d2b2.slice/crio-0d259c735a33d10415e68650e9d0fb0f119e36fbb4fade0277230716d1f8ab4f WatchSource:0}: Error finding container 0d259c735a33d10415e68650e9d0fb0f119e36fbb4fade0277230716d1f8ab4f: Status 404 returned error can't find the container with id 0d259c735a33d10415e68650e9d0fb0f119e36fbb4fade0277230716d1f8ab4f
Sep 04 22:10:49.389747 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.389710    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:49.487453 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.487425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:49.499206 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.499172    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ssvlz" event={"ID":"2ca6195d-72ed-44c3-bc9e-449566276bfe","Type":"ContainerStarted","Data":"deae7c27b7055574d9a9e6a33231bc8bed0609096438092a7d1d566445c305f8"}
Sep 04 22:10:49.499599 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.499234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ssvlz" event={"ID":"2ca6195d-72ed-44c3-bc9e-449566276bfe","Type":"ContainerStarted","Data":"8250d3db2cc7faa449063dc8f8ff4c07a3f67d466cf6bb3501a20f83124b24a8"}
Sep 04 22:10:49.503352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.503047    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwnlx" event={"ID":"b1f78268-1b3c-4799-9e9c-c30b0a297b6c","Type":"ContainerStarted","Data":"688de19f66836b371584017aa4bd62b514fe6e8529282cc432ffa6f3d3abb0c8"}
Sep 04 22:10:49.503352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.503126    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-lwnlx" event={"ID":"b1f78268-1b3c-4799-9e9c-c30b0a297b6c","Type":"ContainerStarted","Data":"6e7474beaf0ca0e4ddb1c2e02490b62717efcb336d5c75526ff978b5ab8a0598"}
Sep 04 22:10:49.505256 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.505234    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerID="bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8" exitCode=0
Sep 04 22:10:49.505352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.505315    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerDied","Data":"bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8"}
Sep 04 22:10:49.507652 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.507630    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6" event={"ID":"7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1","Type":"ContainerStarted","Data":"0133b250595fe7be28cdbdeccf058a70d8ae1f5e8e6a813ef65aec25c6b98ff6"}
Sep 04 22:10:49.508957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.508932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l" event={"ID":"d92d9363-9172-4535-8eba-c8402302d2b2","Type":"ContainerStarted","Data":"0d259c735a33d10415e68650e9d0fb0f119e36fbb4fade0277230716d1f8ab4f"}
Sep 04 22:10:49.516973 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.516922    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ssvlz" podStartSLOduration=131.305140517 podStartE2EDuration="2m12.516905556s" totalImagesPullingTime="1.211765039s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:08:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:49.516628963 +0000 UTC m=+133.035351545" watchObservedRunningTime="2026-09-04 22:10:49.516905556 +0000 UTC m=+133.035628079"
Sep 04 22:10:49.535160 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.535123    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-5jsh6" podStartSLOduration=1.988464381 podStartE2EDuration="3.535109787s" totalImagesPullingTime="1.546645406s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:49.533580782 +0000 UTC m=+133.052303315" watchObservedRunningTime="2026-09-04 22:10:49.535109787 +0000 UTC m=+133.053832320"
Sep 04 22:10:49.582972 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.582918    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-lwnlx" podStartSLOduration=2.489135444 podStartE2EDuration="3.582901712s" totalImagesPullingTime="1.001747394s" totalInitContainerRuntime="92.018874ms" isStatefulPod=true podCreationTimestamp="2026-09-04 22:10:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:49.582003112 +0000 UTC m=+133.100725646" watchObservedRunningTime="2026-09-04 22:10:49.582901712 +0000 UTC m=+133.101624244"
Sep 04 22:10:49.618896 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.618852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:49.722788 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.722761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:49.822479 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.822442    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:49.912981 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:49.912951    2576 scope.go:117] "RemoveContainer" containerID="e3a7a9f1c5e937bedf694adf37df366b7d50ff7b78c4c43a3c73c7b63425968a"
Sep 04 22:10:50.522930 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.522879    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:10:50.523987 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.523962    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-85b454874b-mj6ng" event={"ID":"cb72a802-417c-458b-bca5-c291adeb9bd1","Type":"ContainerStarted","Data":"61094e237614ee4eaa6c4a15d70d6f2823fd87ca8203343c92b6d35577bff81c"}
Sep 04 22:10:50.524440 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.524421    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:50.808246 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.808152    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"]
Sep 04 22:10:50.812046 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.812013    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.815199 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.814960    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Sep 04 22:10:50.815199 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.814986    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 04 22:10:50.816733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.816508    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-mnlgr\""
Sep 04 22:10:50.816733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.816569    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Sep 04 22:10:50.816733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.816511    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Sep 04 22:10:50.817300 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.817114    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-b8dpvshiunt1u\""
Sep 04 22:10:50.823405 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.823367    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"]
Sep 04 22:10:50.850871 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.850838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-secret-metrics-server-client-certs\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.851031 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.850902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-secret-metrics-server-tls\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.851031 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.850927    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-audit-log\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.851031 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.850977    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ks5td\" (UniqueName: \"kubernetes.io/projected/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-kube-api-access-ks5td\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.851229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.851039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.851229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.851108    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-metrics-server-audit-profiles\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.851229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.851165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-client-ca-bundle\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.952331 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.952285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ks5td\" (UniqueName: \"kubernetes.io/projected/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-kube-api-access-ks5td\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.952512 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.952353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.952512 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.952416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-metrics-server-audit-profiles\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.952512 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.952453    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-client-ca-bundle\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.952512 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.952496    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-secret-metrics-server-client-certs\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.952737 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.952551    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-secret-metrics-server-tls\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.952737 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.952575    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-audit-log\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.952929 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.952900    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-audit-log\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.953308 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.953282    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.953570 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.953514    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-metrics-server-audit-profiles\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.955693 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.955665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-secret-metrics-server-client-certs\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.955797 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.955772    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-secret-metrics-server-tls\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.956221 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.956202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-client-ca-bundle\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:50.962195 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:50.962170    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ks5td\" (UniqueName: \"kubernetes.io/projected/d5bc3346-6eda-44bb-8f0c-7fe984ed48f3-kube-api-access-ks5td\") pod \"metrics-server-7fcccdd6d8-sdvlw\" (UID: \"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3\") " pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:51.124793 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.124707    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:10:51.152403 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.152365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:51.154042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.154011    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"]
Sep 04 22:10:51.157287 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.157270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:51.159810 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.159789    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 04 22:10:51.160101 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.160043    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-p9ctz\""
Sep 04 22:10:51.165939 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.165915    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"]
Sep 04 22:10:51.255914 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.255879    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1b7b90a-1a0b-4350-9b40-73b0781e48f5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fpj5j\" (UID: \"e1b7b90a-1a0b-4350-9b40-73b0781e48f5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:51.256124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.255937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zntld\" (UniqueName: \"kubernetes.io/projected/e1b7b90a-1a0b-4350-9b40-73b0781e48f5-kube-api-access-zntld\") pod \"monitoring-plugin-85bbbbbf4-fpj5j\" (UID: \"e1b7b90a-1a0b-4350-9b40-73b0781e48f5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:51.272231 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.272198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:51.328233 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.328195    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-85b454874b-mj6ng"
Sep 04 22:10:51.357014 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.356855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1b7b90a-1a0b-4350-9b40-73b0781e48f5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fpj5j\" (UID: \"e1b7b90a-1a0b-4350-9b40-73b0781e48f5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:51.357014 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.356911    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zntld\" (UniqueName: \"kubernetes.io/projected/e1b7b90a-1a0b-4350-9b40-73b0781e48f5-kube-api-access-zntld\") pod \"monitoring-plugin-85bbbbbf4-fpj5j\" (UID: \"e1b7b90a-1a0b-4350-9b40-73b0781e48f5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:51.361039 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.360093    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/e1b7b90a-1a0b-4350-9b40-73b0781e48f5-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-fpj5j\" (UID: \"e1b7b90a-1a0b-4350-9b40-73b0781e48f5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:51.367852 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.367796    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zntld\" (UniqueName: \"kubernetes.io/projected/e1b7b90a-1a0b-4350-9b40-73b0781e48f5-kube-api-access-zntld\") pod \"monitoring-plugin-85bbbbbf4-fpj5j\" (UID: \"e1b7b90a-1a0b-4350-9b40-73b0781e48f5\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:51.468142 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.468033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:51.541041 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.541007    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:51.642677 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.642571    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:51.836676 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.836598    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:51.887961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.887852    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"]
Sep 04 22:10:51.889422 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:51.889402    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"]
Sep 04 22:10:51.890741 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:51.890702    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5bc3346_6eda_44bb_8f0c_7fe984ed48f3.slice/crio-d241f8521e48169067ce2ba7a026fb64fad4bfd0bc131549814fed8b62ffa11d WatchSource:0}: Error finding container d241f8521e48169067ce2ba7a026fb64fad4bfd0bc131549814fed8b62ffa11d: Status 404 returned error can't find the container with id d241f8521e48169067ce2ba7a026fb64fad4bfd0bc131549814fed8b62ffa11d
Sep 04 22:10:51.900005 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:51.899968    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b7b90a_1a0b_4350_9b40_73b0781e48f5.slice/crio-1d722a2c27feedb530eeb6b72964ea677e649993a16678ebf5dd4cf177ba3876 WatchSource:0}: Error finding container 1d722a2c27feedb530eeb6b72964ea677e649993a16678ebf5dd4cf177ba3876: Status 404 returned error can't find the container with id 1d722a2c27feedb530eeb6b72964ea677e649993a16678ebf5dd4cf177ba3876
Sep 04 22:10:52.193321 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.193245    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:52.266920 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.266887    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:52.538601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.538450    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l" event={"ID":"d92d9363-9172-4535-8eba-c8402302d2b2","Type":"ContainerStarted","Data":"cb41634c1d3cebf71358988566b7ab80d18a5949d18d7b34e3c2e3ad4dcb7dbe"}
Sep 04 22:10:52.538601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.538543    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l" event={"ID":"d92d9363-9172-4535-8eba-c8402302d2b2","Type":"ContainerStarted","Data":"268c6eceb24a6ced1cf7e99cb175393b8eb13934162955fbfdbd151b9b097406"}
Sep 04 22:10:52.538601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.538588    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l" event={"ID":"d92d9363-9172-4535-8eba-c8402302d2b2","Type":"ContainerStarted","Data":"89a78d572aa7ebce5dee89ee316f9b3348ae9922d6a53c49c2d0275888c2d22b"}
Sep 04 22:10:52.543365 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.543332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerStarted","Data":"1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074"}
Sep 04 22:10:52.543774 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.543430    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerStarted","Data":"a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301"}
Sep 04 22:10:52.543774 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.543474    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerStarted","Data":"77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa"}
Sep 04 22:10:52.543774 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.543513    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerStarted","Data":"ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3"}
Sep 04 22:10:52.543774 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.543551    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerStarted","Data":"779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c"}
Sep 04 22:10:52.545761 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.545734    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j" event={"ID":"e1b7b90a-1a0b-4350-9b40-73b0781e48f5","Type":"ContainerStarted","Data":"1d722a2c27feedb530eeb6b72964ea677e649993a16678ebf5dd4cf177ba3876"}
Sep 04 22:10:52.548048 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.547965    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw" event={"ID":"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3","Type":"ContainerStarted","Data":"d241f8521e48169067ce2ba7a026fb64fad4bfd0bc131549814fed8b62ffa11d"}
Sep 04 22:10:52.657260 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.657224    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:52.683786 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.683752    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 04 22:10:52.688355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.688327    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.691428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.691198    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 04 22:10:52.691428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.691303    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 04 22:10:52.691609 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.691519    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 04 22:10:52.691913 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.691759    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 04 22:10:52.693668 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.692314    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 04 22:10:52.693668 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.692637    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 04 22:10:52.693668 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.693422    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 04 22:10:52.693668 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.693459    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 04 22:10:52.694698 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.694350    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 04 22:10:52.694698 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.694370    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 04 22:10:52.694698 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.694534    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-747fluqulobsq\""
Sep 04 22:10:52.694698 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.694663    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 04 22:10:52.694904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.694821    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-z97hb\""
Sep 04 22:10:52.694904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.694848    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 04 22:10:52.694988 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.694973    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 04 22:10:52.699023 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.699001    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 04 22:10:52.707511 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.707482    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 04 22:10:52.750554 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.750471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:52.772338 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772308    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-config-out\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772352    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772375    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772418    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772434    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772458    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772487    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772537    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-config\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772563    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772591    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772608    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.772654 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772649    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.773149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-web-config\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.773149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lst6\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-kube-api-access-6lst6\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.773149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772730    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.773149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772744    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.773149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.773149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.772812    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874186 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874157    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874320 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874200    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874320 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874225    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874320 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874246    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-config\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874530 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874480    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874530 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874512    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874627 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874556    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874627 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874581    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874627 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874603    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-web-config\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874660    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lst6\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-kube-api-access-6lst6\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874694    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874727    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874980 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874980 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-config-out\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874980 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874857    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874980 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.874980 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.876019 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874983    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.876019 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.875007    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.876019 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.874985    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.876019 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.875863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.877022 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.876962    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.877270 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.877245    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.879118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.878758    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.879118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.879022    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-web-config\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.880949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.879469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-config\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.880949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.879773    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.880949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.880326    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.880949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.880861    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.881234 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.881178    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-config-out\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.881815 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.881786    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.881935 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.881910    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.883019 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.882459    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.883555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.883528    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.885920 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.885813    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.885920 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.885852    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.886208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.886085    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:52.889161 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:52.889119    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lst6\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-kube-api-access-6lst6\") pod \"prometheus-k8s-0\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:53.005883 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.005838    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:10:53.048422 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.048319    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:53.146875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.146815    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:53.409092 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.408978    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:53.556067 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.556027    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l" event={"ID":"d92d9363-9172-4535-8eba-c8402302d2b2","Type":"ContainerStarted","Data":"0f4f4a9cbf0ede0714fa9cf1dc647385c7fdab02630948d7969de269f2d74571"}
Sep 04 22:10:53.556511 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.556155    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l" event={"ID":"d92d9363-9172-4535-8eba-c8402302d2b2","Type":"ContainerStarted","Data":"d3ce34db825807a99624c86353fe91f1556b51c1b676596200d8e200a6eebaad"}
Sep 04 22:10:53.561995 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.561904    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerStarted","Data":"f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52"}
Sep 04 22:10:53.593737 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.593624    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.43426558 podStartE2EDuration="6.593603619s" totalImagesPullingTime="4.141013704s" totalInitContainerRuntime="18.324335ms" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:53.592144395 +0000 UTC m=+137.110866931" watchObservedRunningTime="2026-09-04 22:10:53.593603619 +0000 UTC m=+137.112326153"
Sep 04 22:10:53.613638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.613602    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:53.717480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.717451    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:53.955347 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:53.955314    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 04 22:10:53.958524 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:10:53.958496    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d506bbf_9e4d_4515_aceb_820f75d60342.slice/crio-144d7355b8ee7065ece89afcbc2d8c71ab7513bb46413e3da44b169efa87bc9e WatchSource:0}: Error finding container 144d7355b8ee7065ece89afcbc2d8c71ab7513bb46413e3da44b169efa87bc9e: Status 404 returned error can't find the container with id 144d7355b8ee7065ece89afcbc2d8c71ab7513bb46413e3da44b169efa87bc9e
Sep 04 22:10:54.569520 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.569481    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l" event={"ID":"d92d9363-9172-4535-8eba-c8402302d2b2","Type":"ContainerStarted","Data":"40cf0e96bf8dde1a3c048a6834aaecf401ae2f66695453f6168ab6fa097818fd"}
Sep 04 22:10:54.569975 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.569596    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:10:54.570981 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.570956    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j" event={"ID":"e1b7b90a-1a0b-4350-9b40-73b0781e48f5","Type":"ContainerStarted","Data":"7a3210a23e6f7b0cbc46e1b7162c84f3a5465ef964f2184e0a5cea13a57b608b"}
Sep 04 22:10:54.571121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.571090    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:54.572620 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.572596    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw" event={"ID":"d5bc3346-6eda-44bb-8f0c-7fe984ed48f3","Type":"ContainerStarted","Data":"e2ca163a4be5281b15941f0f43bbac3c89a7cdd5876431d19536f82bafff4bab"}
Sep 04 22:10:54.574338 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.574317    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerID="a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc" exitCode=0
Sep 04 22:10:54.574434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.574359    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerDied","Data":"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"}
Sep 04 22:10:54.574434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.574425    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerStarted","Data":"144d7355b8ee7065ece89afcbc2d8c71ab7513bb46413e3da44b169efa87bc9e"}
Sep 04 22:10:54.576460 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.576442    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j"
Sep 04 22:10:54.597094 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.597018    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l" podStartSLOduration=3.200377929 podStartE2EDuration="6.596997324s" totalImagesPullingTime="3.396619395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:48 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:54.592630529 +0000 UTC m=+138.111353062" watchObservedRunningTime="2026-09-04 22:10:54.596997324 +0000 UTC m=+138.115719860"
Sep 04 22:10:54.611242 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.611195    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw" podStartSLOduration=2.819881282 podStartE2EDuration="4.611180651s" totalImagesPullingTime="1.791299369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:54.610187323 +0000 UTC m=+138.128909858" watchObservedRunningTime="2026-09-04 22:10:54.611180651 +0000 UTC m=+138.129903183"
Sep 04 22:10:54.625740 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:54.625697    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-fpj5j" podStartSLOduration=1.843956559 podStartE2EDuration="3.625683614s" totalImagesPullingTime="1.781727055s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:54.625100309 +0000 UTC m=+138.143822843" watchObservedRunningTime="2026-09-04 22:10:54.625683614 +0000 UTC m=+138.144406148"
Sep 04 22:10:55.026410 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:55.026364    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:56.431940 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:56.431909    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:10:57.338518 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:57.338484    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:57.434657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:57.434610    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:57.526535 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:57.526499    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:58.002354 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:58.002324    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:58.103383 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:58.103324    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:58.596139 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:58.596106    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerStarted","Data":"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"}
Sep 04 22:10:58.596642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:58.596189    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerStarted","Data":"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"}
Sep 04 22:10:58.596642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:58.596237    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerStarted","Data":"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"}
Sep 04 22:10:58.596642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:58.596279    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerStarted","Data":"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"}
Sep 04 22:10:58.596642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:58.596326    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerStarted","Data":"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"}
Sep 04 22:10:58.596642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:58.596362    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerStarted","Data":"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"}
Sep 04 22:10:58.632041 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:58.631973    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.493173447 podStartE2EDuration="6.631950644s" totalImagesPullingTime="3.120841144s" totalInitContainerRuntime="17.936053ms" isStatefulPod=false podCreationTimestamp="2026-09-04 22:10:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:10:58.630778292 +0000 UTC m=+142.149500830" watchObservedRunningTime="2026-09-04 22:10:58.631950644 +0000 UTC m=+142.150673180"
Sep 04 22:10:59.417733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:59.417702    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:10:59.518545 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:10:59.518511    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:00.586814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:00.586781    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-57498c5b8c-4j56l"
Sep 04 22:11:01.453805 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.453761    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8" podUID="bc67c976-3d0c-458a-bcc9-4f7fc06b085d" containerName="registry" containerID="cri-o://d47ff6cea0edc1f83974f49134dbadefb01da1344bc800ff7eb84978f3120126" gracePeriod=30
Sep 04 22:11:01.611846 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.611811    2576 generic.go:356] "Generic (PLEG): container finished" podID="bc67c976-3d0c-458a-bcc9-4f7fc06b085d" containerID="d47ff6cea0edc1f83974f49134dbadefb01da1344bc800ff7eb84978f3120126" exitCode=0
Sep 04 22:11:01.612251 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.611888    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8" event={"ID":"bc67c976-3d0c-458a-bcc9-4f7fc06b085d","Type":"ContainerDied","Data":"d47ff6cea0edc1f83974f49134dbadefb01da1344bc800ff7eb84978f3120126"}
Sep 04 22:11:01.699515 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.699492    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:11:01.760918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.760834    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-image-registry-private-configuration\") pod \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") "
Sep 04 22:11:01.760918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.760900    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-ca-trust-extracted\") pod \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") "
Sep 04 22:11:01.761148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.760936    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-installation-pull-secrets\") pod \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") "
Sep 04 22:11:01.761148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.760957    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78zn4\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-kube-api-access-78zn4\") pod \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") "
Sep 04 22:11:01.761148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.760979    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") pod \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") "
Sep 04 22:11:01.761148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.761006    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-trusted-ca\") pod \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") "
Sep 04 22:11:01.761148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.761039    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-certificates\") pod \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") "
Sep 04 22:11:01.761148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.761082    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-bound-sa-token\") pod \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\" (UID: \"bc67c976-3d0c-458a-bcc9-4f7fc06b085d\") "
Sep 04 22:11:01.761523 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.761492    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bc67c976-3d0c-458a-bcc9-4f7fc06b085d" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:11:01.761594 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.761568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bc67c976-3d0c-458a-bcc9-4f7fc06b085d" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:11:01.761873 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.761848    2576 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-trusted-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:11:01.761985 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.761877    2576 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-certificates\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:11:01.763615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.763583    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bc67c976-3d0c-458a-bcc9-4f7fc06b085d" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:11:01.763707 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.763649    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bc67c976-3d0c-458a-bcc9-4f7fc06b085d" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:11:01.763707 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.763664    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-kube-api-access-78zn4" (OuterVolumeSpecName: "kube-api-access-78zn4") pod "bc67c976-3d0c-458a-bcc9-4f7fc06b085d" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d"). InnerVolumeSpecName "kube-api-access-78zn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:11:01.763851 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.763833    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bc67c976-3d0c-458a-bcc9-4f7fc06b085d" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:11:01.763898 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.763852    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bc67c976-3d0c-458a-bcc9-4f7fc06b085d" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:11:01.769650 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.769627    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bc67c976-3d0c-458a-bcc9-4f7fc06b085d" (UID: "bc67c976-3d0c-458a-bcc9-4f7fc06b085d"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:11:01.862777 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.862739    2576 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-ca-trust-extracted\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:11:01.862777 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.862770    2576 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-installation-pull-secrets\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:11:01.862777 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.862780    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78zn4\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-kube-api-access-78zn4\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:11:01.863004 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.862791    2576 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-registry-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:11:01.863004 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.862800    2576 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-bound-sa-token\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:11:01.863004 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:01.862809    2576 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bc67c976-3d0c-458a-bcc9-4f7fc06b085d-image-registry-private-configuration\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:11:02.616877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:02.616849    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8"
Sep 04 22:11:02.616877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:02.616870    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-8d54ffd4b-bs4n8" event={"ID":"bc67c976-3d0c-458a-bcc9-4f7fc06b085d","Type":"ContainerDied","Data":"fb737e45e642eb8a2cb19b7f838ac11b5bbd9af08c9255abb5ea901251e839d7"}
Sep 04 22:11:02.617408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:02.616926    2576 scope.go:117] "RemoveContainer" containerID="d47ff6cea0edc1f83974f49134dbadefb01da1344bc800ff7eb84978f3120126"
Sep 04 22:11:02.646314 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:02.646286    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-8d54ffd4b-bs4n8"]
Sep 04 22:11:02.655482 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:02.655458    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-8d54ffd4b-bs4n8"]
Sep 04 22:11:02.918622 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:02.918541    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc67c976-3d0c-458a-bcc9-4f7fc06b085d" path="/var/lib/kubelet/pods/bc67c976-3d0c-458a-bcc9-4f7fc06b085d/volumes"
Sep 04 22:11:03.007009 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:03.006971    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:11:04.190042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:04.190005    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:04.283240 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:04.283194    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:07.086310 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:07.086267    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:07.187102 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:07.187069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:08.195485 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:08.195434    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:08.265118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:08.265083    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:09.023911 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:09.023869    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:09.131029 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:09.130997    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:11.125025 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:11.124988    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:11:11.125428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:11.125078    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:11:11.229286 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:11.229247    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:11.290930 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:11.290900    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:13.847521 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:11:13.847478    2576 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-79rjk" podUID="2faabeaa-73ea-4a77-b707-765541109e4d"
Sep 04 22:11:14.670922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:14.670886    2576 generic.go:356] "Generic (PLEG): container finished" podID="85c7d188-f68c-4717-884d-fbf00586d73d" containerID="7f9883ceba36abc1ad986a736d3d12f7762370543684719aa6b749b7486516bd" exitCode=0
Sep 04 22:11:14.671196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:14.670961    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk" event={"ID":"85c7d188-f68c-4717-884d-fbf00586d73d","Type":"ContainerDied","Data":"7f9883ceba36abc1ad986a736d3d12f7762370543684719aa6b749b7486516bd"}
Sep 04 22:11:14.671196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:14.671027    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-79rjk"
Sep 04 22:11:14.671494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:14.671477    2576 scope.go:117] "RemoveContainer" containerID="7f9883ceba36abc1ad986a736d3d12f7762370543684719aa6b749b7486516bd"
Sep 04 22:11:15.455395 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:15.455360    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:15.555882 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:15.555848    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:15.657236 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:15.657198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:15.677395 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:15.677354    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-58wlk" event={"ID":"85c7d188-f68c-4717-884d-fbf00586d73d","Type":"ContainerStarted","Data":"5956c9d84989943b0edc3c4cfd36293aaf12f316bd81a2f4676ce26933fe62a9"}
Sep 04 22:11:17.079768 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:17.079737    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6c0849a-f082-4085-9e32-42f6f5ea75e6/init-config-reloader/0.log"
Sep 04 22:11:17.277351 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:17.277306    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:17.279397 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:17.279373    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6c0849a-f082-4085-9e32-42f6f5ea75e6/alertmanager/0.log"
Sep 04 22:11:17.336248 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:17.336149    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:17.480299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:17.480268    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6c0849a-f082-4085-9e32-42f6f5ea75e6/config-reloader/0.log"
Sep 04 22:11:17.680537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:17.680455    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6c0849a-f082-4085-9e32-42f6f5ea75e6/kube-rbac-proxy-web/0.log"
Sep 04 22:11:17.880511 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:17.880471    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6c0849a-f082-4085-9e32-42f6f5ea75e6/kube-rbac-proxy/0.log"
Sep 04 22:11:18.080388 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:18.080351    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6c0849a-f082-4085-9e32-42f6f5ea75e6/kube-rbac-proxy-metric/0.log"
Sep 04 22:11:18.279145 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:18.279117    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_c6c0849a-f082-4085-9e32-42f6f5ea75e6/prom-label-proxy/0.log"
Sep 04 22:11:18.482286 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:18.482254    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-q4jht_1fbbc3a8-c52a-4592-b802-1b64b5c2a23f/cluster-monitoring-operator/0.log"
Sep 04 22:11:18.811103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:18.811027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:11:18.813370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:18.813349    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2faabeaa-73ea-4a77-b707-765541109e4d-metrics-tls\") pod \"dns-default-79rjk\" (UID: \"2faabeaa-73ea-4a77-b707-765541109e4d\") " pod="openshift-dns/dns-default-79rjk"
Sep 04 22:11:18.874692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:18.874654    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wcrtk\""
Sep 04 22:11:18.882743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:18.882722    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-79rjk"
Sep 04 22:11:19.157327 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:19.157259    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-79rjk"]
Sep 04 22:11:19.161184 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:11:19.161157    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2faabeaa_73ea_4a77_b707_765541109e4d.slice/crio-e81dd54eac4613f7cf7feebc07388193a917c58d0ef17418d3209915f6df8ffd WatchSource:0}: Error finding container e81dd54eac4613f7cf7feebc07388193a917c58d0ef17418d3209915f6df8ffd: Status 404 returned error can't find the container with id e81dd54eac4613f7cf7feebc07388193a917c58d0ef17418d3209915f6df8ffd
Sep 04 22:11:19.279770 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:19.279739    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fcccdd6d8-sdvlw_d5bc3346-6eda-44bb-8f0c-7fe984ed48f3/metrics-server/0.log"
Sep 04 22:11:19.480168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:19.480136    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-fpj5j_e1b7b90a-1a0b-4350-9b40-73b0781e48f5/monitoring-plugin/0.log"
Sep 04 22:11:19.646285 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:19.646235    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:19.696490 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:19.696450    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-79rjk" event={"ID":"2faabeaa-73ea-4a77-b707-765541109e4d","Type":"ContainerStarted","Data":"e81dd54eac4613f7cf7feebc07388193a917c58d0ef17418d3209915f6df8ffd"}
Sep 04 22:11:20.281605 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:20.281574    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwnlx_b1f78268-1b3c-4799-9e9c-c30b0a297b6c/init-textfile/0.log"
Sep 04 22:11:20.481902 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:20.481877    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwnlx_b1f78268-1b3c-4799-9e9c-c30b0a297b6c/node-exporter/0.log"
Sep 04 22:11:20.546762 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:20.546542    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:20.637120 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:20.637083    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:20.680083 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:20.680037    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwnlx_b1f78268-1b3c-4799-9e9c-c30b0a297b6c/kube-rbac-proxy/0.log"
Sep 04 22:11:21.479355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:21.479315    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5jsh6_7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1/kube-rbac-proxy-main/0.log"
Sep 04 22:11:21.680097 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:21.680044    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5jsh6_7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1/kube-rbac-proxy-self/0.log"
Sep 04 22:11:21.707036 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:21.707006    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-79rjk" event={"ID":"2faabeaa-73ea-4a77-b707-765541109e4d","Type":"ContainerStarted","Data":"e3b2711c6c62a589179dfdb1df3e08d63f08fc2ea8fb942558e3bc62f422fc80"}
Sep 04 22:11:21.707186 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:21.707050    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-79rjk" event={"ID":"2faabeaa-73ea-4a77-b707-765541109e4d","Type":"ContainerStarted","Data":"b544040b84279709a23715f98a3011cb5a30a53098a0cd477af8994f60e8824d"}
Sep 04 22:11:21.707249 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:21.707201    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-79rjk"
Sep 04 22:11:21.727026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:21.726989    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-79rjk" podStartSLOduration=129.902140311 podStartE2EDuration="2m11.726974789s" totalImagesPullingTime="1.824834478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:09:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:11:21.725532936 +0000 UTC m=+165.244255461" watchObservedRunningTime="2026-09-04 22:11:21.726974789 +0000 UTC m=+165.245697321"
Sep 04 22:11:21.880262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:21.880176    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5jsh6_7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1/openshift-state-metrics/0.log"
Sep 04 22:11:22.080681 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:22.080629    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5d506bbf-9e4d-4515-aceb-820f75d60342/init-config-reloader/0.log"
Sep 04 22:11:22.281599 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:22.281562    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5d506bbf-9e4d-4515-aceb-820f75d60342/prometheus/0.log"
Sep 04 22:11:22.480567 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:22.480538    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5d506bbf-9e4d-4515-aceb-820f75d60342/config-reloader/0.log"
Sep 04 22:11:22.680337 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:22.680244    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5d506bbf-9e4d-4515-aceb-820f75d60342/thanos-sidecar/0.log"
Sep 04 22:11:22.880696 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:22.880665    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5d506bbf-9e4d-4515-aceb-820f75d60342/kube-rbac-proxy-web/0.log"
Sep 04 22:11:23.080810 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:23.080780    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5d506bbf-9e4d-4515-aceb-820f75d60342/kube-rbac-proxy/0.log"
Sep 04 22:11:23.201861 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:23.201824    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:23.271859 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:23.271820    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:23.279743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:23.279718    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_5d506bbf-9e4d-4515-aceb-820f75d60342/kube-rbac-proxy-thanos/0.log"
Sep 04 22:11:23.879893 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:23.879867    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-962mh_c996da1e-c340-4079-bec0-af1f00551466/prometheus-operator-admission-webhook/0.log"
Sep 04 22:11:24.080564 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:24.080531    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/thanos-query/0.log"
Sep 04 22:11:24.280626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:24.280597    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/kube-rbac-proxy-web/0.log"
Sep 04 22:11:24.479643 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:24.479610    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/kube-rbac-proxy/0.log"
Sep 04 22:11:24.681031 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:24.680957    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/prom-label-proxy/0.log"
Sep 04 22:11:24.880777 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:24.880748    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/kube-rbac-proxy-rules/0.log"
Sep 04 22:11:25.080960 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:25.080921    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/kube-rbac-proxy-metrics/0.log"
Sep 04 22:11:25.279989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:25.279960    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-qt6pg_4e302d63-80ad-4370-8feb-17b2bf538ee2/networking-console-plugin/0.log"
Sep 04 22:11:25.479874 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:25.479847    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:11:25.683541 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:25.683500    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/3.log"
Sep 04 22:11:26.493358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:26.493331    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-79rjk_2faabeaa-73ea-4a77-b707-765541109e4d/dns/0.log"
Sep 04 22:11:26.680383 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:26.680357    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-79rjk_2faabeaa-73ea-4a77-b707-765541109e4d/kube-rbac-proxy/0.log"
Sep 04 22:11:27.680252 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:27.680219    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5mt7k_862daba0-07cf-499c-b35f-6a0d34c49cc9/dns-node-resolver/0.log"
Sep 04 22:11:28.481342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:28.481314    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-76mz2_f6434544-e7c3-4235-850c-3d4471c2c030/serve-healthcheck-canary/0.log"
Sep 04 22:11:31.137322 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:31.137291    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:11:31.141492 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:31.141468    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7fcccdd6d8-sdvlw"
Sep 04 22:11:31.714373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:31.714345    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-79rjk"
Sep 04 22:11:33.292956 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:33.292911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:40.880537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:40.880500    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:40.980995 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:40.980950    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:11:53.007148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:53.007102    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:11:53.047123 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:53.047091    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:11:53.861321 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:11:53.861280    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:00.589466 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:00.589422    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:05.722665 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:05.722624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:06.215935 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:06.215894    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:06.796230 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:06.796196    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 04 22:12:06.796640 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:06.796517    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="alertmanager" containerID="cri-o://779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c" gracePeriod=120
Sep 04 22:12:06.796640 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:06.796582    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy-web" containerID="cri-o://77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa" gracePeriod=120
Sep 04 22:12:06.796640 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:06.796611    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="prom-label-proxy" containerID="cri-o://f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52" gracePeriod=120
Sep 04 22:12:06.796822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:06.796611    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy" containerID="cri-o://a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301" gracePeriod=120
Sep 04 22:12:06.796822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:06.796629    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="config-reloader" containerID="cri-o://ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3" gracePeriod=120
Sep 04 22:12:06.796822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:06.796568    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy-metric" containerID="cri-o://1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074" gracePeriod=120
Sep 04 22:12:07.909858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:07.909508    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerID="f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52" exitCode=0
Sep 04 22:12:07.909858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:07.909529    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerID="a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301" exitCode=0
Sep 04 22:12:07.909858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:07.909535    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerID="ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3" exitCode=0
Sep 04 22:12:07.909858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:07.909541    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerID="779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c" exitCode=0
Sep 04 22:12:07.909858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:07.909591    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerDied","Data":"f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52"}
Sep 04 22:12:07.909858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:07.909682    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerDied","Data":"a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301"}
Sep 04 22:12:07.909858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:07.909726    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerDied","Data":"ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3"}
Sep 04 22:12:07.909858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:07.909766    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerDied","Data":"779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c"}
Sep 04 22:12:08.041474 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.041447    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:08.151276 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151195    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-tls-assets\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.151276 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151238    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk7lt\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-kube-api-access-rk7lt\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.151276 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151261    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-trusted-ca-bundle\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.151539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151292    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-out\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.151539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151332    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-web\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.151539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151369    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-volume\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.151539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151401    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-metric\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.151539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151452    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-web-config\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.151813 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151775    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:12:08.151933 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151911    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-metrics-client-ca\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.152013 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151950    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-main-tls\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.152013 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.151983    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.152147 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.152029    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-main-db\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.152147 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.152074    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-cluster-tls-config\") pod \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\" (UID: \"c6c0849a-f082-4085-9e32-42f6f5ea75e6\") "
Sep 04 22:12:08.152394 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.152373    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.153020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.152979    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:12:08.153020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.153006    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:12:08.153891 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.153863    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-kube-api-access-rk7lt" (OuterVolumeSpecName: "kube-api-access-rk7lt") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "kube-api-access-rk7lt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:12:08.154598 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.154571    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:08.155036 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.155005    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-out" (OuterVolumeSpecName: "config-out") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:12:08.155466 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.155438    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-volume" (OuterVolumeSpecName: "config-volume") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:08.155563 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.155539    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:08.155667 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.155645    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:08.155727 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.155704    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:08.155942 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.155927    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:12:08.158589 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.158563    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:08.165974 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.165953    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-web-config" (OuterVolumeSpecName: "web-config") pod "c6c0849a-f082-4085-9e32-42f6f5ea75e6" (UID: "c6c0849a-f082-4085-9e32-42f6f5ea75e6"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:08.254270 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254229    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6c0849a-f082-4085-9e32-42f6f5ea75e6-metrics-client-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254270 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254264    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-main-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254270 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254275    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254285    2576 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-alertmanager-main-db\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254295    2576 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-cluster-tls-config\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254304    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-tls-assets\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254312    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk7lt\" (UniqueName: \"kubernetes.io/projected/c6c0849a-f082-4085-9e32-42f6f5ea75e6-kube-api-access-rk7lt\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254321    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-out\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254330    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254338    2576 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-config-volume\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254346    2576 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.254499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.254356    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/c6c0849a-f082-4085-9e32-42f6f5ea75e6-web-config\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:08.916786 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.916756    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerID="1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074" exitCode=0
Sep 04 22:12:08.916786 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.916781    2576 generic.go:356] "Generic (PLEG): container finished" podID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerID="77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa" exitCode=0
Sep 04 22:12:08.917206 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.916891    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:08.917663 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.917629    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerDied","Data":"1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074"}
Sep 04 22:12:08.917758 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.917700    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerDied","Data":"77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa"}
Sep 04 22:12:08.917758 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.917743    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"c6c0849a-f082-4085-9e32-42f6f5ea75e6","Type":"ContainerDied","Data":"2b8d88f03904d85fbf7eac02a22f19b30d9de494cf86b0eab6b62312aab77438"}
Sep 04 22:12:08.917830 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.917775    2576 scope.go:117] "RemoveContainer" containerID="f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52"
Sep 04 22:12:08.926103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.926074    2576 scope.go:117] "RemoveContainer" containerID="1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074"
Sep 04 22:12:08.933403 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.933383    2576 scope.go:117] "RemoveContainer" containerID="a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301"
Sep 04 22:12:08.940284 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.940266    2576 scope.go:117] "RemoveContainer" containerID="77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa"
Sep 04 22:12:08.947120 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.947096    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 04 22:12:08.947877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.947863    2576 scope.go:117] "RemoveContainer" containerID="ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3"
Sep 04 22:12:08.951210 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.951184    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 04 22:12:08.955704 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.955687    2576 scope.go:117] "RemoveContainer" containerID="779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c"
Sep 04 22:12:08.962896 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.962879    2576 scope.go:117] "RemoveContainer" containerID="bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8"
Sep 04 22:12:08.969980 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.969961    2576 scope.go:117] "RemoveContainer" containerID="f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52"
Sep 04 22:12:08.970275 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:08.970252    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52\": container with ID starting with f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52 not found: ID does not exist" containerID="f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52"
Sep 04 22:12:08.970324 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.970283    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52"} err="failed to get container status \"f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52\": rpc error: code = NotFound desc = could not find container \"f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52\": container with ID starting with f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52 not found: ID does not exist"
Sep 04 22:12:08.970324 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.970316    2576 scope.go:117] "RemoveContainer" containerID="1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074"
Sep 04 22:12:08.970570 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:08.970554    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074\": container with ID starting with 1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074 not found: ID does not exist" containerID="1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074"
Sep 04 22:12:08.970610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.970576    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074"} err="failed to get container status \"1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074\": rpc error: code = NotFound desc = could not find container \"1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074\": container with ID starting with 1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074 not found: ID does not exist"
Sep 04 22:12:08.970610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.970589    2576 scope.go:117] "RemoveContainer" containerID="a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301"
Sep 04 22:12:08.970853 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:08.970821    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301\": container with ID starting with a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301 not found: ID does not exist" containerID="a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301"
Sep 04 22:12:08.970894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.970859    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301"} err="failed to get container status \"a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301\": rpc error: code = NotFound desc = could not find container \"a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301\": container with ID starting with a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301 not found: ID does not exist"
Sep 04 22:12:08.970894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.970877    2576 scope.go:117] "RemoveContainer" containerID="77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa"
Sep 04 22:12:08.971156 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:08.971138    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa\": container with ID starting with 77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa not found: ID does not exist" containerID="77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa"
Sep 04 22:12:08.971209 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.971160    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa"} err="failed to get container status \"77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa\": rpc error: code = NotFound desc = could not find container \"77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa\": container with ID starting with 77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa not found: ID does not exist"
Sep 04 22:12:08.971209 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.971177    2576 scope.go:117] "RemoveContainer" containerID="ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3"
Sep 04 22:12:08.971412 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:08.971395    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3\": container with ID starting with ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3 not found: ID does not exist" containerID="ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3"
Sep 04 22:12:08.971455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.971415    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3"} err="failed to get container status \"ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3\": rpc error: code = NotFound desc = could not find container \"ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3\": container with ID starting with ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3 not found: ID does not exist"
Sep 04 22:12:08.971455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.971428    2576 scope.go:117] "RemoveContainer" containerID="779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c"
Sep 04 22:12:08.971630 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:08.971614    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c\": container with ID starting with 779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c not found: ID does not exist" containerID="779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c"
Sep 04 22:12:08.971665 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.971632    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c"} err="failed to get container status \"779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c\": rpc error: code = NotFound desc = could not find container \"779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c\": container with ID starting with 779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c not found: ID does not exist"
Sep 04 22:12:08.971665 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.971643    2576 scope.go:117] "RemoveContainer" containerID="bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8"
Sep 04 22:12:08.971826 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:08.971810    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8\": container with ID starting with bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8 not found: ID does not exist" containerID="bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8"
Sep 04 22:12:08.971866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.971829    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8"} err="failed to get container status \"bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8\": rpc error: code = NotFound desc = could not find container \"bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8\": container with ID starting with bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8 not found: ID does not exist"
Sep 04 22:12:08.971866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.971840    2576 scope.go:117] "RemoveContainer" containerID="f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52"
Sep 04 22:12:08.972042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972021    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52"} err="failed to get container status \"f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52\": rpc error: code = NotFound desc = could not find container \"f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52\": container with ID starting with f0bf8151c7dafef6b7e6140125e7e862d7f261bdd6e00256c3e956d6a51fba52 not found: ID does not exist"
Sep 04 22:12:08.972042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972036    2576 scope.go:117] "RemoveContainer" containerID="1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074"
Sep 04 22:12:08.972276 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972257    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074"} err="failed to get container status \"1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074\": rpc error: code = NotFound desc = could not find container \"1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074\": container with ID starting with 1db72a272e0c560361d22b8477485bd36ba7d5cf749e1f623eb4f8c8f97a6074 not found: ID does not exist"
Sep 04 22:12:08.972316 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972277    2576 scope.go:117] "RemoveContainer" containerID="a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301"
Sep 04 22:12:08.972475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972457    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301"} err="failed to get container status \"a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301\": rpc error: code = NotFound desc = could not find container \"a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301\": container with ID starting with a437bbb3d436b1be1e0030b5d0780a95b51e72f730ab085625f8e8c19544f301 not found: ID does not exist"
Sep 04 22:12:08.972517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972475    2576 scope.go:117] "RemoveContainer" containerID="77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa"
Sep 04 22:12:08.972709 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972687    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa"} err="failed to get container status \"77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa\": rpc error: code = NotFound desc = could not find container \"77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa\": container with ID starting with 77ba296aedbad6da33a5aee14a6af27578fb0b6b03bb1df5a6b0fb8b84a3bbaa not found: ID does not exist"
Sep 04 22:12:08.972781 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972711    2576 scope.go:117] "RemoveContainer" containerID="ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3"
Sep 04 22:12:08.972912 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972897    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3"} err="failed to get container status \"ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3\": rpc error: code = NotFound desc = could not find container \"ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3\": container with ID starting with ce7d341dd38bd9974ab8c4f87929bac8fca0dd1e3b4864f06845a0ae85cd49e3 not found: ID does not exist"
Sep 04 22:12:08.972960 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.972911    2576 scope.go:117] "RemoveContainer" containerID="779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c"
Sep 04 22:12:08.973164 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.973147    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c"} err="failed to get container status \"779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c\": rpc error: code = NotFound desc = could not find container \"779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c\": container with ID starting with 779d6caccb10ae069a495e613e2cbab77d2d97c3a6204982a2a956ae2d4e977c not found: ID does not exist"
Sep 04 22:12:08.973204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.973165    2576 scope.go:117] "RemoveContainer" containerID="bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8"
Sep 04 22:12:08.973352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.973334    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8"} err="failed to get container status \"bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8\": rpc error: code = NotFound desc = could not find container \"bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8\": container with ID starting with bc2bd7f4991bfa46d7d40c8209f44ae7d5a08494b26963087045d49f48026eb8 not found: ID does not exist"
Sep 04 22:12:08.978330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978306    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 04 22:12:08.978648 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978632    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy"
Sep 04 22:12:08.978721 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978651    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy"
Sep 04 22:12:08.978721 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978669    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="alertmanager"
Sep 04 22:12:08.978721 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978679    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="alertmanager"
Sep 04 22:12:08.978721 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978709    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy-metric"
Sep 04 22:12:08.978721 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978718    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy-metric"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978734    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="init-config-reloader"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978743    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="init-config-reloader"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978755    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="config-reloader"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978764    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="config-reloader"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978773    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy-web"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978781    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy-web"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978792    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="prom-label-proxy"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978800    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="prom-label-proxy"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978810    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc67c976-3d0c-458a-bcc9-4f7fc06b085d" containerName="registry"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978818    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc67c976-3d0c-458a-bcc9-4f7fc06b085d" containerName="registry"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978900    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy-web"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978916    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="alertmanager"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978925    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy-metric"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978935    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="config-reloader"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978944    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="prom-label-proxy"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978955    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc67c976-3d0c-458a-bcc9-4f7fc06b085d" containerName="registry"
Sep 04 22:12:08.978967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.978963    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" containerName="kube-rbac-proxy"
Sep 04 22:12:08.984399 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.984380    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:08.987417 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.987382    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 04 22:12:08.987499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.987382    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 04 22:12:08.987499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.987489    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 04 22:12:08.987657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.987439    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 04 22:12:08.987657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.987514    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 04 22:12:08.987657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.987514    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 04 22:12:08.987870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.987791    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 04 22:12:08.987870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.987835    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wls69\""
Sep 04 22:12:08.987870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.987864    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 04 22:12:08.991972 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.991950    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 04 22:12:08.996874 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:08.996853    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 04 22:12:09.161924 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.161838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.161924 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.161882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.161924 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.161902    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162158 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.161979    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162158 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.162023    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162158 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.162045    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-web-config\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162158 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.162122    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-config-out\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162280 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.162166    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162280 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.162187    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95pxn\" (UniqueName: \"kubernetes.io/projected/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-kube-api-access-95pxn\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162280 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.162208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.162282    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.162303    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.162371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.162328    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.262860 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.262818    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-config-out\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.262879    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.262900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95pxn\" (UniqueName: \"kubernetes.io/projected/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-kube-api-access-95pxn\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.262921    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.262972    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.262999    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.263035    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.263089    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.263132    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.263161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.263198    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.263227    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.263252    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-web-config\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.263551 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.263329    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.264081 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.264033    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.264626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.264301    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.266025 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.265997    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.266170 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.266148    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-web-config\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.266242 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.266193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-config-volume\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.266242 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.266225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-config-out\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.266413 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.266389    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.266537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.266522    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.266609 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.266588    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.267159 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.267139    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.267989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.267968    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.272686 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.272668    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95pxn\" (UniqueName: \"kubernetes.io/projected/f1fda9de-0d40-446f-bbb8-e52ac79fe2f3-kube-api-access-95pxn\") pod \"alertmanager-main-0\" (UID: \"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.293738 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.293715    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 04 22:12:09.576804 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.576766    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 04 22:12:09.577864 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:12:09.577841    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1fda9de_0d40_446f_bbb8_e52ac79fe2f3.slice/crio-85b5524e56395f870b519eddf973fcb1fd237b35f37fcf7acfd22f90fff68798 WatchSource:0}: Error finding container 85b5524e56395f870b519eddf973fcb1fd237b35f37fcf7acfd22f90fff68798: Status 404 returned error can't find the container with id 85b5524e56395f870b519eddf973fcb1fd237b35f37fcf7acfd22f90fff68798
Sep 04 22:12:09.924991 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.924904    2576 generic.go:356] "Generic (PLEG): container finished" podID="f1fda9de-0d40-446f-bbb8-e52ac79fe2f3" containerID="e4aa3ff44e34784bfa36b0f66d05ebefa1ced0cf52a292a384b461f17482857f" exitCode=0
Sep 04 22:12:09.924991 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.924948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3","Type":"ContainerDied","Data":"e4aa3ff44e34784bfa36b0f66d05ebefa1ced0cf52a292a384b461f17482857f"}
Sep 04 22:12:09.925418 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:09.925000    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3","Type":"ContainerStarted","Data":"85b5524e56395f870b519eddf973fcb1fd237b35f37fcf7acfd22f90fff68798"}
Sep 04 22:12:10.829292 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.829251    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"]
Sep 04 22:12:10.832896 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.832872    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:10.835613 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.835584    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 04 22:12:10.835730 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.835611    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 04 22:12:10.835730 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.835612    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9jbv6\""
Sep 04 22:12:10.835934 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.835918    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 04 22:12:10.835979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.835961    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 04 22:12:10.839218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.839198    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 04 22:12:10.843154 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.843136    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 04 22:12:10.846957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.846937    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"]
Sep 04 22:12:10.919711 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.919665    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c0849a-f082-4085-9e32-42f6f5ea75e6" path="/var/lib/kubelet/pods/c6c0849a-f082-4085-9e32-42f6f5ea75e6/volumes"
Sep 04 22:12:10.931559 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.931526    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3","Type":"ContainerStarted","Data":"92bb60c46c0a41a54dab3057d8e672fee6dd1249e1e419bd14cada303d956444"}
Sep 04 22:12:10.931894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.931594    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3","Type":"ContainerStarted","Data":"1eb131f2573c1e4db3be34658c947177b51390cbdd2a640c8b86f6c84fd346fb"}
Sep 04 22:12:10.931894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.931625    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3","Type":"ContainerStarted","Data":"d227e06832974caf1c6ecc1ab15f2cdefffbc294d8494f10250f4a628b72f07d"}
Sep 04 22:12:10.931894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.931650    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3","Type":"ContainerStarted","Data":"c38d1e5c515d57b706023a37252e6bf9e1e7fbf5addeec90d4eb0023005f9a1b"}
Sep 04 22:12:10.931894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.931681    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3","Type":"ContainerStarted","Data":"431c6207dc90891846da707654703485a84acc525ff33b5581d0124ff0643781"}
Sep 04 22:12:10.931894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.931717    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f1fda9de-0d40-446f-bbb8-e52ac79fe2f3","Type":"ContainerStarted","Data":"a6148eeae48f1f082d9a48278ef132f4511d1729833642d40b37a798cbc8e9de"}
Sep 04 22:12:10.962389 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.962326    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.9474530899999998 podStartE2EDuration="2.962307383s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.854293ms" isStatefulPod=false podCreationTimestamp="2026-09-04 22:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:12:10.961045933 +0000 UTC m=+214.479768502" watchObservedRunningTime="2026-09-04 22:12:10.962307383 +0000 UTC m=+214.481029916"
Sep 04 22:12:10.979240 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.979201    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szb4s\" (UniqueName: \"kubernetes.io/projected/3bf4d2ce-1024-4505-b2cf-33187bfc6540-kube-api-access-szb4s\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:10.979393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.979266    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf4d2ce-1024-4505-b2cf-33187bfc6540-serving-certs-ca-bundle\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:10.979393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.979343    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4d2ce-1024-4505-b2cf-33187bfc6540-metrics-client-ca\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:10.979471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.979397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf4d2ce-1024-4505-b2cf-33187bfc6540-telemeter-trusted-ca-bundle\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:10.979503 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.979480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:10.979537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.979514    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-secret-telemeter-client\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:10.979581 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.979540    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-telemeter-client-tls\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:10.979660 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:10.979640    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-federate-client-tls\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.080880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.080787    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4d2ce-1024-4505-b2cf-33187bfc6540-metrics-client-ca\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.080880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.080826    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf4d2ce-1024-4505-b2cf-33187bfc6540-telemeter-trusted-ca-bundle\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.081157 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.081042    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.081157 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.081113    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-secret-telemeter-client\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.081271 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.081155    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-telemeter-client-tls\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.081271 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.081251    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-federate-client-tls\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.081369 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.081353    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szb4s\" (UniqueName: \"kubernetes.io/projected/3bf4d2ce-1024-4505-b2cf-33187bfc6540-kube-api-access-szb4s\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.081420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.081394    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf4d2ce-1024-4505-b2cf-33187bfc6540-serving-certs-ca-bundle\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.081703 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.081671    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3bf4d2ce-1024-4505-b2cf-33187bfc6540-metrics-client-ca\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.081824 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.081737    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf4d2ce-1024-4505-b2cf-33187bfc6540-telemeter-trusted-ca-bundle\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.082277 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.082253    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3bf4d2ce-1024-4505-b2cf-33187bfc6540-serving-certs-ca-bundle\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.083692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.083671    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-telemeter-client-tls\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.083865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.083843    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-secret-telemeter-client\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.083935 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.083918    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.083986 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.083961    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/3bf4d2ce-1024-4505-b2cf-33187bfc6540-federate-client-tls\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.091354 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.091330    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szb4s\" (UniqueName: \"kubernetes.io/projected/3bf4d2ce-1024-4505-b2cf-33187bfc6540-kube-api-access-szb4s\") pod \"telemeter-client-87dbd8d9f-25kvp\" (UID: \"3bf4d2ce-1024-4505-b2cf-33187bfc6540\") " pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.137857 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.137824    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 04 22:12:11.138130 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.138100    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="prometheus" containerID="cri-o://339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18" gracePeriod=600
Sep 04 22:12:11.138268 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.138131    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy-thanos" containerID="cri-o://cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff" gracePeriod=600
Sep 04 22:12:11.138268 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.138149    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy-web" containerID="cri-o://f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de" gracePeriod=600
Sep 04 22:12:11.138268 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.138180    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy" containerID="cri-o://732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877" gracePeriod=600
Sep 04 22:12:11.138268 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.138194    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="config-reloader" containerID="cri-o://7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338" gracePeriod=600
Sep 04 22:12:11.138268 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.138170    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="thanos-sidecar" containerID="cri-o://834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250" gracePeriod=600
Sep 04 22:12:11.143428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.142504    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"
Sep 04 22:12:11.376632 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.376608    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:11.433576 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.433542    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-87dbd8d9f-25kvp"]
Sep 04 22:12:11.437744 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:12:11.437713    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bf4d2ce_1024_4505_b2cf_33187bfc6540.slice/crio-bcb37017fea97133a3d327330573017176ca57db26e96518dfd7c03b1a793fc5 WatchSource:0}: Error finding container bcb37017fea97133a3d327330573017176ca57db26e96518dfd7c03b1a793fc5: Status 404 returned error can't find the container with id bcb37017fea97133a3d327330573017176ca57db26e96518dfd7c03b1a793fc5
Sep 04 22:12:11.484393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484365    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-tls-assets\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-thanos-prometheus-http-client-file\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484423    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-web-config\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484441    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-config\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484469    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-config-out\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484520    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-serving-certs-ca-bundle\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484566    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484593    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-metrics-client-ca\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484623    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-kubelet-serving-ca-bundle\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484648    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-2\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484718    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-grpc-tls\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484744    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484773    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-trusted-ca-bundle\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.484865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484828    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-1\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.485196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484869    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lst6\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-kube-api-access-6lst6\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.485196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484898    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-tls\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.485196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484926    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-kube-rbac-proxy\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.485196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484959    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-0\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.485196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.484998    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-db\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.485196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.485021    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-metrics-client-certs\") pod \"5d506bbf-9e4d-4515-aceb-820f75d60342\" (UID: \"5d506bbf-9e4d-4515-aceb-820f75d60342\") "
Sep 04 22:12:11.485496 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.485325    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:12:11.485750 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.485718    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:12:11.486989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.486957    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:12:11.487103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.486984    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:12:11.487176 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.487115    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-config-out" (OuterVolumeSpecName: "config-out") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:12:11.487233 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.487188    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:12:11.487233 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.487198    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:12:11.487345 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.487330    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:12:11.487782 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.487756    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:12:11.488362 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.488227    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:11.488362 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.488333    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:11.489049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.488651    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:12:11.489049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.488987    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:11.489049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.489006    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-config" (OuterVolumeSpecName: "config") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:11.489049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.489016    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:11.489640 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.489623    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:11.489874 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.489855    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:11.490103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.490086    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-kube-api-access-6lst6" (OuterVolumeSpecName: "kube-api-access-6lst6") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "kube-api-access-6lst6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:12:11.490164 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.490094    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:11.498656 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.498636    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-web-config" (OuterVolumeSpecName: "web-config") pod "5d506bbf-9e4d-4515-aceb-820f75d60342" (UID: "5d506bbf-9e4d-4515-aceb-820f75d60342"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:12:11.586377 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586343    2576 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-kube-rbac-proxy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586377 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586369    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586377 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586379    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-db\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586387    2576 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-metrics-client-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586396    2576 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-tls-assets\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586404    2576 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-thanos-prometheus-http-client-file\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586415    2576 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-web-config\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586423    2576 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-config\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586430    2576 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5d506bbf-9e4d-4515-aceb-820f75d60342-config-out\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586438    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586446    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586456    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-metrics-client-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586466    2576 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586475    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586483    2576 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-grpc-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586495    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586508    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-trusted-ca-bundle\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586518    2576 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/5d506bbf-9e4d-4515-aceb-820f75d60342-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586527    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lst6\" (UniqueName: \"kubernetes.io/projected/5d506bbf-9e4d-4515-aceb-820f75d60342-kube-api-access-6lst6\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.586649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.586536    2576 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/5d506bbf-9e4d-4515-aceb-820f75d60342-secret-prometheus-k8s-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:12:11.939517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939481    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerID="cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff" exitCode=0
Sep 04 22:12:11.939517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939513    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerID="732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877" exitCode=0
Sep 04 22:12:11.939517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939523    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerID="f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de" exitCode=0
Sep 04 22:12:11.939999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939531    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerID="834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250" exitCode=0
Sep 04 22:12:11.939999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939539    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerID="7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338" exitCode=0
Sep 04 22:12:11.939999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939547    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerID="339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18" exitCode=0
Sep 04 22:12:11.939999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939565    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerDied","Data":"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"}
Sep 04 22:12:11.939999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939633    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerDied","Data":"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"}
Sep 04 22:12:11.939999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939574    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:11.939999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939673    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerDied","Data":"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"}
Sep 04 22:12:11.939999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.939712    2576 scope.go:117] "RemoveContainer" containerID="cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"
Sep 04 22:12:11.940376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.940132    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerDied","Data":"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"}
Sep 04 22:12:11.940376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.940235    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerDied","Data":"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"}
Sep 04 22:12:11.940376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.940294    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerDied","Data":"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"}
Sep 04 22:12:11.940376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.940348    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"5d506bbf-9e4d-4515-aceb-820f75d60342","Type":"ContainerDied","Data":"144d7355b8ee7065ece89afcbc2d8c71ab7513bb46413e3da44b169efa87bc9e"}
Sep 04 22:12:11.945698 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.945527    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp" event={"ID":"3bf4d2ce-1024-4505-b2cf-33187bfc6540","Type":"ContainerStarted","Data":"bcb37017fea97133a3d327330573017176ca57db26e96518dfd7c03b1a793fc5"}
Sep 04 22:12:11.953320 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.953303    2576 scope.go:117] "RemoveContainer" containerID="732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"
Sep 04 22:12:11.959998 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.959983    2576 scope.go:117] "RemoveContainer" containerID="f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"
Sep 04 22:12:11.966311 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.966294    2576 scope.go:117] "RemoveContainer" containerID="834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"
Sep 04 22:12:11.972218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.972197    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 04 22:12:11.973415 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.973402    2576 scope.go:117] "RemoveContainer" containerID="7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"
Sep 04 22:12:11.979718 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.979695    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 04 22:12:11.980441 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.980426    2576 scope.go:117] "RemoveContainer" containerID="339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"
Sep 04 22:12:11.987453 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.987438    2576 scope.go:117] "RemoveContainer" containerID="a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"
Sep 04 22:12:11.993981 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.993963    2576 scope.go:117] "RemoveContainer" containerID="cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"
Sep 04 22:12:11.994247 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:11.994230    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": container with ID starting with cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff not found: ID does not exist" containerID="cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"
Sep 04 22:12:11.994301 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.994256    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"} err="failed to get container status \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": rpc error: code = NotFound desc = could not find container \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": container with ID starting with cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff not found: ID does not exist"
Sep 04 22:12:11.994301 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.994273    2576 scope.go:117] "RemoveContainer" containerID="732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"
Sep 04 22:12:11.994498 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:11.994481    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": container with ID starting with 732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877 not found: ID does not exist" containerID="732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"
Sep 04 22:12:11.994534 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.994501    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"} err="failed to get container status \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": rpc error: code = NotFound desc = could not find container \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": container with ID starting with 732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877 not found: ID does not exist"
Sep 04 22:12:11.994534 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.994514    2576 scope.go:117] "RemoveContainer" containerID="f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"
Sep 04 22:12:11.994719 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:11.994695    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": container with ID starting with f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de not found: ID does not exist" containerID="f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"
Sep 04 22:12:11.994803 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.994723    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"} err="failed to get container status \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": rpc error: code = NotFound desc = could not find container \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": container with ID starting with f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de not found: ID does not exist"
Sep 04 22:12:11.994803 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.994735    2576 scope.go:117] "RemoveContainer" containerID="834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"
Sep 04 22:12:11.994960 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:11.994945    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": container with ID starting with 834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250 not found: ID does not exist" containerID="834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"
Sep 04 22:12:11.995002 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.994964    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"} err="failed to get container status \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": rpc error: code = NotFound desc = could not find container \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": container with ID starting with 834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250 not found: ID does not exist"
Sep 04 22:12:11.995002 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.994976    2576 scope.go:117] "RemoveContainer" containerID="7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"
Sep 04 22:12:11.995292 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:11.995277    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": container with ID starting with 7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338 not found: ID does not exist" containerID="7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"
Sep 04 22:12:11.995332 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.995296    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"} err="failed to get container status \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": rpc error: code = NotFound desc = could not find container \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": container with ID starting with 7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338 not found: ID does not exist"
Sep 04 22:12:11.995332 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.995310    2576 scope.go:117] "RemoveContainer" containerID="339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"
Sep 04 22:12:11.995559 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:11.995541    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": container with ID starting with 339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18 not found: ID does not exist" containerID="339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"
Sep 04 22:12:11.995612 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.995564    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"} err="failed to get container status \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": rpc error: code = NotFound desc = could not find container \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": container with ID starting with 339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18 not found: ID does not exist"
Sep 04 22:12:11.995612 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.995579    2576 scope.go:117] "RemoveContainer" containerID="a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"
Sep 04 22:12:11.995828 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:12:11.995812    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": container with ID starting with a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc not found: ID does not exist" containerID="a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"
Sep 04 22:12:11.995867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.995839    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"} err="failed to get container status \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": rpc error: code = NotFound desc = could not find container \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": container with ID starting with a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc not found: ID does not exist"
Sep 04 22:12:11.995867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.995853    2576 scope.go:117] "RemoveContainer" containerID="cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"
Sep 04 22:12:11.996073 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996042    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"} err="failed to get container status \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": rpc error: code = NotFound desc = could not find container \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": container with ID starting with cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff not found: ID does not exist"
Sep 04 22:12:11.996129 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996074    2576 scope.go:117] "RemoveContainer" containerID="732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"
Sep 04 22:12:11.996312 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996295    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"} err="failed to get container status \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": rpc error: code = NotFound desc = could not find container \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": container with ID starting with 732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877 not found: ID does not exist"
Sep 04 22:12:11.996355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996312    2576 scope.go:117] "RemoveContainer" containerID="f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"
Sep 04 22:12:11.996539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996523    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"} err="failed to get container status \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": rpc error: code = NotFound desc = could not find container \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": container with ID starting with f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de not found: ID does not exist"
Sep 04 22:12:11.996593 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996539    2576 scope.go:117] "RemoveContainer" containerID="834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"
Sep 04 22:12:11.996728 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996710    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"} err="failed to get container status \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": rpc error: code = NotFound desc = could not find container \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": container with ID starting with 834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250 not found: ID does not exist"
Sep 04 22:12:11.996770 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996730    2576 scope.go:117] "RemoveContainer" containerID="7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"
Sep 04 22:12:11.996931 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996913    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"} err="failed to get container status \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": rpc error: code = NotFound desc = could not find container \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": container with ID starting with 7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338 not found: ID does not exist"
Sep 04 22:12:11.996931 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.996928    2576 scope.go:117] "RemoveContainer" containerID="339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"
Sep 04 22:12:11.997158 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997132    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"} err="failed to get container status \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": rpc error: code = NotFound desc = could not find container \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": container with ID starting with 339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18 not found: ID does not exist"
Sep 04 22:12:11.997197 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997158    2576 scope.go:117] "RemoveContainer" containerID="a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"
Sep 04 22:12:11.997389 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997375    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"} err="failed to get container status \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": rpc error: code = NotFound desc = could not find container \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": container with ID starting with a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc not found: ID does not exist"
Sep 04 22:12:11.997431 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997390    2576 scope.go:117] "RemoveContainer" containerID="cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"
Sep 04 22:12:11.997584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997567    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"} err="failed to get container status \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": rpc error: code = NotFound desc = could not find container \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": container with ID starting with cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff not found: ID does not exist"
Sep 04 22:12:11.997624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997584    2576 scope.go:117] "RemoveContainer" containerID="732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"
Sep 04 22:12:11.997787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997774    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"} err="failed to get container status \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": rpc error: code = NotFound desc = could not find container \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": container with ID starting with 732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877 not found: ID does not exist"
Sep 04 22:12:11.997828 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997788    2576 scope.go:117] "RemoveContainer" containerID="f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"
Sep 04 22:12:11.997987 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997972    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"} err="failed to get container status \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": rpc error: code = NotFound desc = could not find container \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": container with ID starting with f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de not found: ID does not exist"
Sep 04 22:12:11.998024 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.997988    2576 scope.go:117] "RemoveContainer" containerID="834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"
Sep 04 22:12:11.998195 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998180    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"} err="failed to get container status \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": rpc error: code = NotFound desc = could not find container \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": container with ID starting with 834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250 not found: ID does not exist"
Sep 04 22:12:11.998244 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998195    2576 scope.go:117] "RemoveContainer" containerID="7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"
Sep 04 22:12:11.998380 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998365    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"} err="failed to get container status \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": rpc error: code = NotFound desc = could not find container \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": container with ID starting with 7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338 not found: ID does not exist"
Sep 04 22:12:11.998427 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998379    2576 scope.go:117] "RemoveContainer" containerID="339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"
Sep 04 22:12:11.998561 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998547    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"} err="failed to get container status \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": rpc error: code = NotFound desc = could not find container \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": container with ID starting with 339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18 not found: ID does not exist"
Sep 04 22:12:11.998605 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998561    2576 scope.go:117] "RemoveContainer" containerID="a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"
Sep 04 22:12:11.998761 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998746    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"} err="failed to get container status \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": rpc error: code = NotFound desc = could not find container \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": container with ID starting with a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc not found: ID does not exist"
Sep 04 22:12:11.998797 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998761    2576 scope.go:117] "RemoveContainer" containerID="cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"
Sep 04 22:12:11.998957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998942    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"} err="failed to get container status \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": rpc error: code = NotFound desc = could not find container \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": container with ID starting with cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff not found: ID does not exist"
Sep 04 22:12:11.999001 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.998957    2576 scope.go:117] "RemoveContainer" containerID="732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"
Sep 04 22:12:11.999189 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.999173    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"} err="failed to get container status \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": rpc error: code = NotFound desc = could not find container \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": container with ID starting with 732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877 not found: ID does not exist"
Sep 04 22:12:11.999189 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.999189    2576 scope.go:117] "RemoveContainer" containerID="f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"
Sep 04 22:12:11.999376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.999359    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"} err="failed to get container status \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": rpc error: code = NotFound desc = could not find container \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": container with ID starting with f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de not found: ID does not exist"
Sep 04 22:12:11.999376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.999375    2576 scope.go:117] "RemoveContainer" containerID="834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"
Sep 04 22:12:11.999631 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.999616    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"} err="failed to get container status \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": rpc error: code = NotFound desc = could not find container \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": container with ID starting with 834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250 not found: ID does not exist"
Sep 04 22:12:11.999631 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.999631    2576 scope.go:117] "RemoveContainer" containerID="7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"
Sep 04 22:12:11.999894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.999867    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"} err="failed to get container status \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": rpc error: code = NotFound desc = could not find container \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": container with ID starting with 7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338 not found: ID does not exist"
Sep 04 22:12:11.999894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:11.999890    2576 scope.go:117] "RemoveContainer" containerID="339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"
Sep 04 22:12:12.000187 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.000164    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"} err="failed to get container status \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": rpc error: code = NotFound desc = could not find container \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": container with ID starting with 339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18 not found: ID does not exist"
Sep 04 22:12:12.000187 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.000185    2576 scope.go:117] "RemoveContainer" containerID="a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"
Sep 04 22:12:12.000433 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.000414    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"} err="failed to get container status \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": rpc error: code = NotFound desc = could not find container \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": container with ID starting with a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc not found: ID does not exist"
Sep 04 22:12:12.000481 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.000433    2576 scope.go:117] "RemoveContainer" containerID="cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"
Sep 04 22:12:12.000707 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.000691    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"} err="failed to get container status \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": rpc error: code = NotFound desc = could not find container \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": container with ID starting with cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff not found: ID does not exist"
Sep 04 22:12:12.000707 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.000707    2576 scope.go:117] "RemoveContainer" containerID="732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"
Sep 04 22:12:12.000948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.000932    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"} err="failed to get container status \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": rpc error: code = NotFound desc = could not find container \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": container with ID starting with 732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877 not found: ID does not exist"
Sep 04 22:12:12.000948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.000947    2576 scope.go:117] "RemoveContainer" containerID="f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"
Sep 04 22:12:12.001148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001131    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"} err="failed to get container status \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": rpc error: code = NotFound desc = could not find container \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": container with ID starting with f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de not found: ID does not exist"
Sep 04 22:12:12.001200 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001148    2576 scope.go:117] "RemoveContainer" containerID="834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"
Sep 04 22:12:12.001354 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001337    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"} err="failed to get container status \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": rpc error: code = NotFound desc = could not find container \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": container with ID starting with 834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250 not found: ID does not exist"
Sep 04 22:12:12.001399 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001354    2576 scope.go:117] "RemoveContainer" containerID="7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"
Sep 04 22:12:12.001544 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001529    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"} err="failed to get container status \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": rpc error: code = NotFound desc = could not find container \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": container with ID starting with 7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338 not found: ID does not exist"
Sep 04 22:12:12.001597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001543    2576 scope.go:117] "RemoveContainer" containerID="339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"
Sep 04 22:12:12.001762 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001747    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"} err="failed to get container status \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": rpc error: code = NotFound desc = could not find container \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": container with ID starting with 339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18 not found: ID does not exist"
Sep 04 22:12:12.001762 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001761    2576 scope.go:117] "RemoveContainer" containerID="a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"
Sep 04 22:12:12.001959 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001944    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"} err="failed to get container status \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": rpc error: code = NotFound desc = could not find container \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": container with ID starting with a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc not found: ID does not exist"
Sep 04 22:12:12.002002 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.001959    2576 scope.go:117] "RemoveContainer" containerID="cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"
Sep 04 22:12:12.002172 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.002157    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff"} err="failed to get container status \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": rpc error: code = NotFound desc = could not find container \"cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff\": container with ID starting with cb0673c4baf66d866a1674f8577cb416f8bbc7995f4c3d39debddf95313eceff not found: ID does not exist"
Sep 04 22:12:12.002245 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.002173    2576 scope.go:117] "RemoveContainer" containerID="732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"
Sep 04 22:12:12.002401 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.002383    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877"} err="failed to get container status \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": rpc error: code = NotFound desc = could not find container \"732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877\": container with ID starting with 732066067e454ed236448b68ba5d6814475ca186b7ee96a378ae314a06276877 not found: ID does not exist"
Sep 04 22:12:12.002469 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.002402    2576 scope.go:117] "RemoveContainer" containerID="f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"
Sep 04 22:12:12.002633 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.002613    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de"} err="failed to get container status \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": rpc error: code = NotFound desc = could not find container \"f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de\": container with ID starting with f4b170e5e79527107654d92408ec835a200ed04ea7a3a3d481c6ed7e757659de not found: ID does not exist"
Sep 04 22:12:12.002681 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.002634    2576 scope.go:117] "RemoveContainer" containerID="834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"
Sep 04 22:12:12.002870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.002853    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250"} err="failed to get container status \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": rpc error: code = NotFound desc = could not find container \"834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250\": container with ID starting with 834ea6ae4388f2ba90580101763d86c2b2cfad7acfd8b9e1328bb7924ec52250 not found: ID does not exist"
Sep 04 22:12:12.002913 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.002870    2576 scope.go:117] "RemoveContainer" containerID="7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"
Sep 04 22:12:12.003098 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.003082    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338"} err="failed to get container status \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": rpc error: code = NotFound desc = could not find container \"7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338\": container with ID starting with 7c8fb9e3d6f9c4b413253b3dab0dba2f70b8eca36059187d3c7a02e839447338 not found: ID does not exist"
Sep 04 22:12:12.003098 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.003097    2576 scope.go:117] "RemoveContainer" containerID="339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"
Sep 04 22:12:12.003327 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.003310    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18"} err="failed to get container status \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": rpc error: code = NotFound desc = could not find container \"339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18\": container with ID starting with 339af0153aad6b32ae5aabed8126e5ec4a9974a2ed8fef3aa7d0abd39e1d6d18 not found: ID does not exist"
Sep 04 22:12:12.003381 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.003327    2576 scope.go:117] "RemoveContainer" containerID="a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"
Sep 04 22:12:12.003565 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.003547    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc"} err="failed to get container status \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": rpc error: code = NotFound desc = could not find container \"a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc\": container with ID starting with a5bf79c8ee18e9da9835779a6bf80d1f28fad1c15ce0aa348c5e469ff0a3aedc not found: ID does not exist"
Sep 04 22:12:12.010735 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.010711    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 04 22:12:12.011021 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011009    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="init-config-reloader"
Sep 04 22:12:12.011074 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011022    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="init-config-reloader"
Sep 04 22:12:12.011074 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011034    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="prometheus"
Sep 04 22:12:12.011074 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011039    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="prometheus"
Sep 04 22:12:12.011074 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011047    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy-web"
Sep 04 22:12:12.011074 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011066    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy-web"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011081    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="config-reloader"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011087    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="config-reloader"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011093    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="thanos-sidecar"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011098    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="thanos-sidecar"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011104    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy-thanos"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011110    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy-thanos"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011120    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011125    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011178    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="prometheus"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011184    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011192    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="thanos-sidecar"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011197    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy-web"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011204    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="config-reloader"
Sep 04 22:12:12.011224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.011211    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" containerName="kube-rbac-proxy-thanos"
Sep 04 22:12:12.016720 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.016704    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.019696 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.019676    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 04 22:12:12.019898 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.019878    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 04 22:12:12.019898 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.019889    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 04 22:12:12.020100 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.019935    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-z97hb\""
Sep 04 22:12:12.020218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.020200    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 04 22:12:12.020279 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.020209    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-747fluqulobsq\""
Sep 04 22:12:12.020279 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.020241    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 04 22:12:12.020584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.020563    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 04 22:12:12.020584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.020581    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 04 22:12:12.020738 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.020575    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 04 22:12:12.020738 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.020574    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 04 22:12:12.021082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.021067    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 04 22:12:12.021236 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.021215    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 04 22:12:12.021564 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.021544    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 04 22:12:12.023317 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.023297    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 04 22:12:12.024896 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.024879    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 04 22:12:12.034190 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.034169    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 04 22:12:12.192500 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192407    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192500 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192500 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192479    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192545    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192587    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192619    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-web-config\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192643    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-config\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70c641ca-9f89-428b-a1c1-057f4b599537-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192751    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.192785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192781    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.193085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192806    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.193085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192830    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.193085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192856    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtrgv\" (UniqueName: \"kubernetes.io/projected/70c641ca-9f89-428b-a1c1-057f4b599537-kube-api-access-jtrgv\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.193085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192919    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70c641ca-9f89-428b-a1c1-057f4b599537-config-out\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.193085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192940    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.193085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.192970    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.193085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.193003    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.193085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.193039    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.193312 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.193121    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.293997 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.293731    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-web-config\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.293997 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.293786    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-config\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.293997 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.293833    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70c641ca-9f89-428b-a1c1-057f4b599537-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.293997 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.293865    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.293997 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.293896    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.293997 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.293926    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.293997 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.293949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.294553 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.293975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.294673 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtrgv\" (UniqueName: \"kubernetes.io/projected/70c641ca-9f89-428b-a1c1-057f4b599537-kube-api-access-jtrgv\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.294673 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294641    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70c641ca-9f89-428b-a1c1-057f4b599537-config-out\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.294673 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294665    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.294858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294720    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.294858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294771    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.294858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294804    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.294858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.295134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294851    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.295134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294917    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.295134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294967    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.295134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.294992    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.295134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.295026    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.295134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.295074    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.295418 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.295327    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.298363 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.298336    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-web-config\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.298473 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.298385    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.298473 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.298408    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.298828 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.298804    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.298902 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.298814    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/70c641ca-9f89-428b-a1c1-057f4b599537-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.299695 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.299328    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.299695 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.299348    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.299695 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.299606    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.299695 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.299657    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-config\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.300401 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.300375    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.300743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.300707    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.300827 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.300789    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.301026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.300972    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.301275 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.301254    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.301925 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.301889    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/70c641ca-9f89-428b-a1c1-057f4b599537-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.302023 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.301937    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/70c641ca-9f89-428b-a1c1-057f4b599537-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.302886 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.302860    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/70c641ca-9f89-428b-a1c1-057f4b599537-config-out\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.307120 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.307099    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtrgv\" (UniqueName: \"kubernetes.io/projected/70c641ca-9f89-428b-a1c1-057f4b599537-kube-api-access-jtrgv\") pod \"prometheus-k8s-0\" (UID: \"70c641ca-9f89-428b-a1c1-057f4b599537\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.327002 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.326955    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:12.620592 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.620543    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 04 22:12:12.623474 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:12:12.623441    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70c641ca_9f89_428b_a1c1_057f4b599537.slice/crio-df3354f613b7a7e5bf0f75dc3219a1ffaa3b08d4f1300c217e85a00580b551d3 WatchSource:0}: Error finding container df3354f613b7a7e5bf0f75dc3219a1ffaa3b08d4f1300c217e85a00580b551d3: Status 404 returned error can't find the container with id df3354f613b7a7e5bf0f75dc3219a1ffaa3b08d4f1300c217e85a00580b551d3
Sep 04 22:12:12.918099 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.918005    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d506bbf-9e4d-4515-aceb-820f75d60342" path="/var/lib/kubelet/pods/5d506bbf-9e4d-4515-aceb-820f75d60342/volumes"
Sep 04 22:12:12.951962 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.951928    2576 generic.go:356] "Generic (PLEG): container finished" podID="70c641ca-9f89-428b-a1c1-057f4b599537" containerID="baa028f5254baab197da4678e5f06200758744b3034dbd38c3257138346a192a" exitCode=0
Sep 04 22:12:12.952411 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.952012    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70c641ca-9f89-428b-a1c1-057f4b599537","Type":"ContainerDied","Data":"baa028f5254baab197da4678e5f06200758744b3034dbd38c3257138346a192a"}
Sep 04 22:12:12.952411 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:12.952122    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70c641ca-9f89-428b-a1c1-057f4b599537","Type":"ContainerStarted","Data":"df3354f613b7a7e5bf0f75dc3219a1ffaa3b08d4f1300c217e85a00580b551d3"}
Sep 04 22:12:13.962249 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.962212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp" event={"ID":"3bf4d2ce-1024-4505-b2cf-33187bfc6540","Type":"ContainerStarted","Data":"a5c4ee7a791278bb58117a1bd40df3fbed07db338dc246f9e562dbf6889de675"}
Sep 04 22:12:13.962751 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.962314    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp" event={"ID":"3bf4d2ce-1024-4505-b2cf-33187bfc6540","Type":"ContainerStarted","Data":"09d0bbb4db1cc8854088405e57920b51a38bc24a18df33f7d5fe0a3bf9a077c1"}
Sep 04 22:12:13.962751 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.962344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp" event={"ID":"3bf4d2ce-1024-4505-b2cf-33187bfc6540","Type":"ContainerStarted","Data":"831985f5fc60f5d46dbdafe64af999ed3d1ef47109bb0132d195b401eb40267d"}
Sep 04 22:12:13.965680 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.965657    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70c641ca-9f89-428b-a1c1-057f4b599537","Type":"ContainerStarted","Data":"47157eedc6ca521c30c6078893ca8a622d9c037d3c2634c93f16c9da50fa840a"}
Sep 04 22:12:13.965779 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.965708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70c641ca-9f89-428b-a1c1-057f4b599537","Type":"ContainerStarted","Data":"c77a240f0431d9a0e15abda4f207f2b0c6e5132e0a754d0cfed4726e2bf96b4a"}
Sep 04 22:12:13.965779 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.965734    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70c641ca-9f89-428b-a1c1-057f4b599537","Type":"ContainerStarted","Data":"ed02e1dea5e20b6cbf75f37871fd84b0341152c03175832e394ae52b0aabe034"}
Sep 04 22:12:13.965779 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.965763    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70c641ca-9f89-428b-a1c1-057f4b599537","Type":"ContainerStarted","Data":"70b92a57c4768aca8506fa231c59ca98391ef85c52dbf1e3d5cf39940bf535b8"}
Sep 04 22:12:13.965885 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.965788    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70c641ca-9f89-428b-a1c1-057f4b599537","Type":"ContainerStarted","Data":"115192343b02c14dd807f10cdca0f7f5c900652057a0935a58e6c7924a1f98dd"}
Sep 04 22:12:13.965885 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.965811    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"70c641ca-9f89-428b-a1c1-057f4b599537","Type":"ContainerStarted","Data":"7cffa8dcaaa554f6a59a5a393c06a0836f7d94fc22bd80e8c2478180dcade83f"}
Sep 04 22:12:13.986776 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:13.986728    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-87dbd8d9f-25kvp" podStartSLOduration=2.066619768 podStartE2EDuration="3.986713034s" totalImagesPullingTime="1.920093266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:12:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:12:13.985372428 +0000 UTC m=+217.504094974" watchObservedRunningTime="2026-09-04 22:12:13.986713034 +0000 UTC m=+217.505435544"
Sep 04 22:12:14.017929 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:14.017870    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.9960958939999998 podStartE2EDuration="3.017838792s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.742898ms" isStatefulPod=false podCreationTimestamp="2026-09-04 22:12:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:12:14.016950089 +0000 UTC m=+217.535672637" watchObservedRunningTime="2026-09-04 22:12:14.017838792 +0000 UTC m=+217.536561337"
Sep 04 22:12:14.144375 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:14.144339    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:14.245278 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:14.245195    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:14.346131 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:14.346079    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:14.877570 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:14.877537    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:14.977554 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:14.977519    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:17.328144 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:17.328100    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:12:24.830914 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:24.830877    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:24.893969 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:24.893934    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:28.088794 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:28.088761    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:30.806508 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:30.806478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:30.871361 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:30.871330    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:45.422976 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:45.422934    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:12:45.490043 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:12:45.490011    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:12.327685 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:12.327588    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:13:12.343902 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:12.343875    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:13:13.224344 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:13.224320    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 04 22:13:20.968581 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:20.968545    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:21.036222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:21.036182    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:32.926813 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:32.926776    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:32.990028 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:32.989974    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:36.830319 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:36.830291    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:13:36.832276 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:36.832249    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:13:36.841962 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:36.841934    2576 kubelet.go:1648] "Image garbage collection succeeded"
Sep 04 22:13:38.210877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:38.210843    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:38.311148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:38.311108    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:43.104970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:43.104933    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:54.130562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:54.130523    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:54.231785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:54.231734    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:13:54.290102 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:13:54.290038    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:14.200181 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:14.200141    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:21.277110 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:21.277070    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:21.371957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:21.371919    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:48.518553 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:48.518518    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:49.872496 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:49.872466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:52.357703 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:52.357672    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:52.463196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:52.463157    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:53.608618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:53.608584    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:53.745921 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:53.745887    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:54.315257 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:54.315218    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:54.514804 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:54.514770    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:56.674023 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:56.673983    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:56.904182 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:56.904145    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:59.166866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:59.166828    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:14:59.275045 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:14:59.275011    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:04.270674 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:04.270636    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:06.049372 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:06.049326    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:06.148770 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:06.148727    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:06.446042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:06.445952    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:06.557474 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:06.557439    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:13.028825 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:13.028782    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:13.193919 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:13.193887    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:14.810871 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:14.810840    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:26.573325 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:26.573281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:26.674063 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:26.674022    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:27.132588 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.132549    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:27.503619 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.503585    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:27.644812 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.644771    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:27.733626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.733595    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"]
Sep 04 22:15:27.737109 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.737090    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.740135 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.740115    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 04 22:15:27.740352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.740333    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 04 22:15:27.741264 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.741226    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 04 22:15:27.741375 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.741266    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-kvsxk\""
Sep 04 22:15:27.742104 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.742085    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 04 22:15:27.756726 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.756674    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"]
Sep 04 22:15:27.807609 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.807570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07170860-e476-40bf-a210-e841ff5297e4-tmp\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.807791 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.807641    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07170860-e476-40bf-a210-e841ff5297e4-manifests\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.807791 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.807690    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8mh8\" (UniqueName: \"kubernetes.io/projected/07170860-e476-40bf-a210-e841ff5297e4-kube-api-access-h8mh8\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.807791 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.807714    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07170860-e476-40bf-a210-e841ff5297e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.807791 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.807736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07170860-e476-40bf-a210-e841ff5297e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.908674 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.908638    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07170860-e476-40bf-a210-e841ff5297e4-manifests\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.908850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.908697    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8mh8\" (UniqueName: \"kubernetes.io/projected/07170860-e476-40bf-a210-e841ff5297e4-kube-api-access-h8mh8\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.908850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.908721    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07170860-e476-40bf-a210-e841ff5297e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.908850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.908741    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07170860-e476-40bf-a210-e841ff5297e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.908850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.908779    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07170860-e476-40bf-a210-e841ff5297e4-tmp\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.909133 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.909107    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/07170860-e476-40bf-a210-e841ff5297e4-manifests\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.910998 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.910968    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/07170860-e476-40bf-a210-e841ff5297e4-tmp\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.911247 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.911229    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/07170860-e476-40bf-a210-e841ff5297e4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.911379 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.911359    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/07170860-e476-40bf-a210-e841ff5297e4-webhook-cert\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:27.936201 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:27.936176    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8mh8\" (UniqueName: \"kubernetes.io/projected/07170860-e476-40bf-a210-e841ff5297e4-kube-api-access-h8mh8\") pod \"opendatahub-operator-controller-manager-cd686f5d7-fh5d7\" (UID: \"07170860-e476-40bf-a210-e841ff5297e4\") " pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:28.041903 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.041818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:28.049644 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.049617    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:28.154901 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.154862    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:28.288452 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.288413    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:28.343910 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.343883    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"]
Sep 04 22:15:28.346196 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:15:28.346169    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07170860_e476_40bf_a210_e841ff5297e4.slice/crio-bf7d39bc5de20f740f25d120a8190c44ae9a320e6e868d574f2c560f590eb0fc WatchSource:0}: Error finding container bf7d39bc5de20f740f25d120a8190c44ae9a320e6e868d574f2c560f590eb0fc: Status 404 returned error can't find the container with id bf7d39bc5de20f740f25d120a8190c44ae9a320e6e868d574f2c560f590eb0fc
Sep 04 22:15:28.347954 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.347937    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 22:15:28.388472 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.388441    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:28.472435 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.472402    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:28.572583 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.572544    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:28.675979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.675886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:28.757387 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:28.757354    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7" event={"ID":"07170860-e476-40bf-a210-e841ff5297e4","Type":"ContainerStarted","Data":"bf7d39bc5de20f740f25d120a8190c44ae9a320e6e868d574f2c560f590eb0fc"}
Sep 04 22:15:31.773918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:31.773877    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7" event={"ID":"07170860-e476-40bf-a210-e841ff5297e4","Type":"ContainerStarted","Data":"c5b9fa91e7d488585c94af64baa1d4352aed3a56b4cbe8150ca006811f37389b"}
Sep 04 22:15:32.780417 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:32.780381    2576 generic.go:356] "Generic (PLEG): container finished" podID="07170860-e476-40bf-a210-e841ff5297e4" containerID="c5b9fa91e7d488585c94af64baa1d4352aed3a56b4cbe8150ca006811f37389b" exitCode=0
Sep 04 22:15:32.780829 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:32.780469    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7" event={"ID":"07170860-e476-40bf-a210-e841ff5297e4","Type":"ContainerDied","Data":"c5b9fa91e7d488585c94af64baa1d4352aed3a56b4cbe8150ca006811f37389b"}
Sep 04 22:15:33.786841 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:33.786805    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7" event={"ID":"07170860-e476-40bf-a210-e841ff5297e4","Type":"ContainerStarted","Data":"e5c148f50ed28f69256c751e74790633e8f6b402b858a10354047f1fbb1673c7"}
Sep 04 22:15:33.787330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:33.786946    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:33.822438 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:33.822383    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7" podStartSLOduration=2.434069525 podStartE2EDuration="6.822365123s" totalImagesPullingTime="3.349556091s" totalInitContainerRuntime="1.038739507s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:15:27 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:15:33.82188664 +0000 UTC m=+417.340609173" watchObservedRunningTime="2026-09-04 22:15:33.822365123 +0000 UTC m=+417.341087658"
Sep 04 22:15:41.805938 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:41.805904    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:41.974640 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:41.974606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:44.793436 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:44.793402    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cd686f5d7-fh5d7"
Sep 04 22:15:45.722307 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.722271    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-868df85875-xl62t"]
Sep 04 22:15:45.732961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.732931    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.736630 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.736598    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 04 22:15:45.736811 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.736680    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-cdg74\""
Sep 04 22:15:45.736811 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.736608    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 04 22:15:45.737075 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.737040    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 04 22:15:45.737162 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.737102    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 04 22:15:45.737369 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.737341    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 04 22:15:45.738706 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.738683    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-868df85875-xl62t"]
Sep 04 22:15:45.868406 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.868373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/db41e725-3569-4528-ac27-448b8508b045-metrics-cert\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.868772 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.868416    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/db41e725-3569-4528-ac27-448b8508b045-manager-config\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.868772 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.868437    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg25p\" (UniqueName: \"kubernetes.io/projected/db41e725-3569-4528-ac27-448b8508b045-kube-api-access-hg25p\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.868772 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.868525    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db41e725-3569-4528-ac27-448b8508b045-cert\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.969763 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.969723    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/db41e725-3569-4528-ac27-448b8508b045-manager-config\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.969763 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.969764    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg25p\" (UniqueName: \"kubernetes.io/projected/db41e725-3569-4528-ac27-448b8508b045-kube-api-access-hg25p\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.970007 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.969817    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db41e725-3569-4528-ac27-448b8508b045-cert\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.970007 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.969894    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/db41e725-3569-4528-ac27-448b8508b045-metrics-cert\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.970525 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.970500    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/db41e725-3569-4528-ac27-448b8508b045-manager-config\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.972441 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.972387    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/db41e725-3569-4528-ac27-448b8508b045-metrics-cert\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.972441 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.972412    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/db41e725-3569-4528-ac27-448b8508b045-cert\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:45.981745 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:45.981718    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg25p\" (UniqueName: \"kubernetes.io/projected/db41e725-3569-4528-ac27-448b8508b045-kube-api-access-hg25p\") pod \"lws-controller-manager-868df85875-xl62t\" (UID: \"db41e725-3569-4528-ac27-448b8508b045\") " pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:46.043648 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:46.043607    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:46.341135 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:46.341042    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-868df85875-xl62t"]
Sep 04 22:15:46.344238 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:15:46.344205    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb41e725_3569_4528_ac27_448b8508b045.slice/crio-29adad960f157e02e05f17fd43ab4ceacda85b2f098356c0dfa331715fd36b96 WatchSource:0}: Error finding container 29adad960f157e02e05f17fd43ab4ceacda85b2f098356c0dfa331715fd36b96: Status 404 returned error can't find the container with id 29adad960f157e02e05f17fd43ab4ceacda85b2f098356c0dfa331715fd36b96
Sep 04 22:15:46.513903 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:46.513857    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:46.796459 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:46.796422    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:46.848743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:46.848708    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t" event={"ID":"db41e725-3569-4528-ac27-448b8508b045","Type":"ContainerStarted","Data":"29adad960f157e02e05f17fd43ab4ceacda85b2f098356c0dfa331715fd36b96"}
Sep 04 22:15:47.391608 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.391565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:47.650308 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.650219    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"]
Sep 04 22:15:47.652811 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.652788    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:47.658741 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.658713    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 04 22:15:47.659297 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.659274    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 04 22:15:47.659595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.659571    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 04 22:15:47.659827 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.659809    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-7sqnp\""
Sep 04 22:15:47.660616 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.660594    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 04 22:15:47.673022 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.672997    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"]
Sep 04 22:15:47.789106 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.789062    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359e04bd-a0b6-4031-bd21-02e24d2452ad-tmp\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:47.789310 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.789182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rndvd\" (UniqueName: \"kubernetes.io/projected/359e04bd-a0b6-4031-bd21-02e24d2452ad-kube-api-access-rndvd\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:47.789383 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.789327    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359e04bd-a0b6-4031-bd21-02e24d2452ad-tls-certs\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:47.889992 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.889956    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rndvd\" (UniqueName: \"kubernetes.io/projected/359e04bd-a0b6-4031-bd21-02e24d2452ad-kube-api-access-rndvd\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:47.890194 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.890047    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359e04bd-a0b6-4031-bd21-02e24d2452ad-tls-certs\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:47.890194 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.890097    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359e04bd-a0b6-4031-bd21-02e24d2452ad-tmp\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:47.890194 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:15:47.890181    2576 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 04 22:15:47.890326 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:15:47.890264    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/359e04bd-a0b6-4031-bd21-02e24d2452ad-tls-certs podName:359e04bd-a0b6-4031-bd21-02e24d2452ad nodeName:}" failed. No retries permitted until 2026-09-04 22:15:48.390244574 +0000 UTC m=+431.908967101 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/359e04bd-a0b6-4031-bd21-02e24d2452ad-tls-certs") pod "kube-auth-proxy-7766956b67-sjbqw" (UID: "359e04bd-a0b6-4031-bd21-02e24d2452ad") : secret "kube-auth-proxy-tls" not found
Sep 04 22:15:47.892321 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.892295    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/359e04bd-a0b6-4031-bd21-02e24d2452ad-tmp\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:47.912415 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:47.912346    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rndvd\" (UniqueName: \"kubernetes.io/projected/359e04bd-a0b6-4031-bd21-02e24d2452ad-kube-api-access-rndvd\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:48.331439 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:48.331392    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:48.396340 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:48.396301    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359e04bd-a0b6-4031-bd21-02e24d2452ad-tls-certs\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:48.399398 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:48.399347    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/359e04bd-a0b6-4031-bd21-02e24d2452ad-tls-certs\") pod \"kube-auth-proxy-7766956b67-sjbqw\" (UID: \"359e04bd-a0b6-4031-bd21-02e24d2452ad\") " pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:48.565906 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:48.565864    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"
Sep 04 22:15:49.164569 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:15:49.164527    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359e04bd_a0b6_4031_bd21_02e24d2452ad.slice/crio-f11694486dee316b24464117a2acbbbf5481b15376303444c41683e92c48efa0 WatchSource:0}: Error finding container f11694486dee316b24464117a2acbbbf5481b15376303444c41683e92c48efa0: Status 404 returned error can't find the container with id f11694486dee316b24464117a2acbbbf5481b15376303444c41683e92c48efa0
Sep 04 22:15:49.170907 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:49.170882    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7766956b67-sjbqw"]
Sep 04 22:15:49.865360 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:49.865313    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw" event={"ID":"359e04bd-a0b6-4031-bd21-02e24d2452ad","Type":"ContainerStarted","Data":"f11694486dee316b24464117a2acbbbf5481b15376303444c41683e92c48efa0"}
Sep 04 22:15:49.867266 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:49.867230    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t" event={"ID":"db41e725-3569-4528-ac27-448b8508b045","Type":"ContainerStarted","Data":"e2b5983e8165977a7fb9693323c9f1f9c9f11e8b4dc6a622016a4ce762e04d91"}
Sep 04 22:15:49.867443 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:49.867418    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:15:49.900323 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:49.899840    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t" podStartSLOduration=2.363447083 podStartE2EDuration="4.899816516s" totalImagesPullingTime="2.536369433s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:15:49.897178467 +0000 UTC m=+433.415900996" watchObservedRunningTime="2026-09-04 22:15:49.899816516 +0000 UTC m=+433.418539050"
Sep 04 22:15:51.010528 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:51.010495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:51.146917 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:51.146873    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:51.878764 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:51.878732    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw" event={"ID":"359e04bd-a0b6-4031-bd21-02e24d2452ad","Type":"ContainerStarted","Data":"623f96214fbf707a5a23f42e6fd3128d83fff8a6376156e672b1a04ec724b990"}
Sep 04 22:15:51.911302 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:51.911249    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7766956b67-sjbqw" podStartSLOduration=2.387359229 podStartE2EDuration="4.911232266s" totalImagesPullingTime="2.523873037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:15:51.910912973 +0000 UTC m=+435.429635507" watchObservedRunningTime="2026-09-04 22:15:51.911232266 +0000 UTC m=+435.429954798"
Sep 04 22:15:52.746685 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:52.746622    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:55.058361 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:55.058325    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:55.160720 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:55.160686    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:55.749943 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:55.749899    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:55.872733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:55.872695    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:55.987442 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:55.987401    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:57.450860 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:57.450813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:57.550315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:57.550276    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:57.934591 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:57.934554    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:58.075297 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:58.075262    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:15:58.213130 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:15:58.213096    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:00.876573 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:00.876494    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-868df85875-xl62t"
Sep 04 22:16:06.084204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:06.084175    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:06.214897 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:06.214859    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:06.728827 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:06.728792    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:06.937021 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:06.936987    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:09.967583 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:09.967542    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:10.215608 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:10.215574    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:10.339485 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:10.339399    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:10.457335 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:10.457295    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:10.557544 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:10.557483    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:17.036339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:17.036305    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:17.185120 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:17.185080    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:22.823551 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:22.823506    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:22.924267 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:22.924236    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:56.622933 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:56.622896    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:58.495411 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:58.495379    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:16:58.574173 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:16:58.574136    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:12.219790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:12.219744    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:13.016439 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:13.016397    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:16.070363 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:16.070326    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:16.161605 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:16.161569    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:16.263981 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:16.263948    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:18.136274 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:18.136239    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:18.238567 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:18.238539    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:18.865416 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:18.865384    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:19.071461 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:19.071418    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:19.225768 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:19.225737    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:19.324491 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:19.324460    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:22.634244 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:22.634201    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:23.298520 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:23.298482    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:24.354475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.354440    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"]
Sep 04 22:17:24.357906 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.357889    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"
Sep 04 22:17:24.361153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.361134    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-h4z5s\""
Sep 04 22:17:24.361263 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.361228    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 04 22:17:24.362379 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.362355    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 04 22:17:24.362379 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.362374    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 04 22:17:24.373310 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.373278    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"]
Sep 04 22:17:24.481624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.481592    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57kqk\" (UniqueName: \"kubernetes.io/projected/db110c23-62e9-4e2f-b655-fa093c21079b-kube-api-access-57kqk\") pod \"dns-operator-controller-manager-5985564776-w4cf5\" (UID: \"db110c23-62e9-4e2f-b655-fa093c21079b\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"
Sep 04 22:17:24.582447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.582416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57kqk\" (UniqueName: \"kubernetes.io/projected/db110c23-62e9-4e2f-b655-fa093c21079b-kube-api-access-57kqk\") pod \"dns-operator-controller-manager-5985564776-w4cf5\" (UID: \"db110c23-62e9-4e2f-b655-fa093c21079b\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"
Sep 04 22:17:24.591737 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.591703    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-57kqk\" (UniqueName: \"kubernetes.io/projected/db110c23-62e9-4e2f-b655-fa093c21079b-kube-api-access-57kqk\") pod \"dns-operator-controller-manager-5985564776-w4cf5\" (UID: \"db110c23-62e9-4e2f-b655-fa093c21079b\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"
Sep 04 22:17:24.668734 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.668640    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"
Sep 04 22:17:24.943914 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:24.943853    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"]
Sep 04 22:17:24.946753 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:17:24.946726    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb110c23_62e9_4e2f_b655_fa093c21079b.slice/crio-aee0d31a3e3bd576c5bb535198bf611a9bd61aa3cfe238b05978a05e11c7de9f WatchSource:0}: Error finding container aee0d31a3e3bd576c5bb535198bf611a9bd61aa3cfe238b05978a05e11c7de9f: Status 404 returned error can't find the container with id aee0d31a3e3bd576c5bb535198bf611a9bd61aa3cfe238b05978a05e11c7de9f
Sep 04 22:17:25.297641 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:25.297608    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5" event={"ID":"db110c23-62e9-4e2f-b655-fa093c21079b","Type":"ContainerStarted","Data":"aee0d31a3e3bd576c5bb535198bf611a9bd61aa3cfe238b05978a05e11c7de9f"}
Sep 04 22:17:25.625441 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:25.625365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:25.724733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:25.724699    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:26.031957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:26.031920    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:26.143362 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:26.143328    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:28.315756 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:28.315712    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5" event={"ID":"db110c23-62e9-4e2f-b655-fa093c21079b","Type":"ContainerStarted","Data":"832feef24dcc4152b17f3527e29e7b5ff39047d04cd794a39f91348cb9ed5d51"}
Sep 04 22:17:28.316191 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:28.315816    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"
Sep 04 22:17:28.335650 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:28.335600    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5" podStartSLOduration=1.446121241 podStartE2EDuration="4.335585517s" totalImagesPullingTime="2.889464276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:17:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:17:28.333965545 +0000 UTC m=+531.852688079" watchObservedRunningTime="2026-09-04 22:17:28.335585517 +0000 UTC m=+531.854308107"
Sep 04 22:17:29.181414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:29.181323    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:29.283968 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:29.283938    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:29.381263 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:29.381226    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:29.481232 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:29.481198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:33.862276 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:33.862234    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:33.960449 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:33.960405    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:34.067164 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:34.067122    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:39.322266 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.322228    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-w4cf5"
Sep 04 22:17:39.702875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.702798    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"]
Sep 04 22:17:39.706229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.706211    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:39.709342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.709320    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-7pnjv\""
Sep 04 22:17:39.715619 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.715601    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d486697-32ca-4b81-ae92-513de62a203c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cqdrc\" (UID: \"5d486697-32ca-4b81-ae92-513de62a203c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:39.715707 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.715659    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t8qn\" (UniqueName: \"kubernetes.io/projected/5d486697-32ca-4b81-ae92-513de62a203c-kube-api-access-9t8qn\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cqdrc\" (UID: \"5d486697-32ca-4b81-ae92-513de62a203c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:39.731833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.731809    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"]
Sep 04 22:17:39.816962 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.816924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d486697-32ca-4b81-ae92-513de62a203c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cqdrc\" (UID: \"5d486697-32ca-4b81-ae92-513de62a203c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:39.817195 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.816997    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t8qn\" (UniqueName: \"kubernetes.io/projected/5d486697-32ca-4b81-ae92-513de62a203c-kube-api-access-9t8qn\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cqdrc\" (UID: \"5d486697-32ca-4b81-ae92-513de62a203c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:39.817381 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.817360    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d486697-32ca-4b81-ae92-513de62a203c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cqdrc\" (UID: \"5d486697-32ca-4b81-ae92-513de62a203c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:39.829180 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:39.829153    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t8qn\" (UniqueName: \"kubernetes.io/projected/5d486697-32ca-4b81-ae92-513de62a203c-kube-api-access-9t8qn\") pod \"kuadrant-operator-controller-manager-6bd956ccf-cqdrc\" (UID: \"5d486697-32ca-4b81-ae92-513de62a203c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:40.016029 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.015991    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:40.303148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.303124    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"]
Sep 04 22:17:40.305428 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:17:40.305400    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d486697_32ca_4b81_ae92_513de62a203c.slice/crio-2851207487c97ba6300f5c035c2d1c2ad13726359f17301433ae80e3c612d519 WatchSource:0}: Error finding container 2851207487c97ba6300f5c035c2d1c2ad13726359f17301433ae80e3c612d519: Status 404 returned error can't find the container with id 2851207487c97ba6300f5c035c2d1c2ad13726359f17301433ae80e3c612d519
Sep 04 22:17:40.370484 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.370448    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc" event={"ID":"5d486697-32ca-4b81-ae92-513de62a203c","Type":"ContainerStarted","Data":"2851207487c97ba6300f5c035c2d1c2ad13726359f17301433ae80e3c612d519"}
Sep 04 22:17:40.417571 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.417535    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"]
Sep 04 22:17:40.431346 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.431310    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:40.435429 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.435403    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"]
Sep 04 22:17:40.530425 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.530389    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"]
Sep 04 22:17:40.534813 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.534790    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"
Sep 04 22:17:40.537835 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.537816    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-p2nbz\""
Sep 04 22:17:40.568776 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.568698    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"]
Sep 04 22:17:40.570776 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.570737    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:40.623476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.623440    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phgt6\" (UniqueName: \"kubernetes.io/projected/a9afdd21-4dfc-4204-b80d-6ad317a02a00-kube-api-access-phgt6\") pod \"limitador-operator-controller-manager-6c549b6d85-7nxmd\" (UID: \"a9afdd21-4dfc-4204-b80d-6ad317a02a00\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"
Sep 04 22:17:40.724621 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.724586    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phgt6\" (UniqueName: \"kubernetes.io/projected/a9afdd21-4dfc-4204-b80d-6ad317a02a00-kube-api-access-phgt6\") pod \"limitador-operator-controller-manager-6c549b6d85-7nxmd\" (UID: \"a9afdd21-4dfc-4204-b80d-6ad317a02a00\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"
Sep 04 22:17:40.739946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.739917    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-phgt6\" (UniqueName: \"kubernetes.io/projected/a9afdd21-4dfc-4204-b80d-6ad317a02a00-kube-api-access-phgt6\") pod \"limitador-operator-controller-manager-6c549b6d85-7nxmd\" (UID: \"a9afdd21-4dfc-4204-b80d-6ad317a02a00\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"
Sep 04 22:17:40.848300 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:40.848201    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"
Sep 04 22:17:41.141400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:41.141307    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"]
Sep 04 22:17:41.144820 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:17:41.144790    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9afdd21_4dfc_4204_b80d_6ad317a02a00.slice/crio-e0f54e384dcdeb06c608774ec3ad814541e8657aaf3dde3e8e6a78f8abc7e68c WatchSource:0}: Error finding container e0f54e384dcdeb06c608774ec3ad814541e8657aaf3dde3e8e6a78f8abc7e68c: Status 404 returned error can't find the container with id e0f54e384dcdeb06c608774ec3ad814541e8657aaf3dde3e8e6a78f8abc7e68c
Sep 04 22:17:41.377656 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:41.377614    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd" event={"ID":"a9afdd21-4dfc-4204-b80d-6ad317a02a00","Type":"ContainerStarted","Data":"e0f54e384dcdeb06c608774ec3ad814541e8657aaf3dde3e8e6a78f8abc7e68c"}
Sep 04 22:17:43.391974 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:43.391928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd" event={"ID":"a9afdd21-4dfc-4204-b80d-6ad317a02a00","Type":"ContainerStarted","Data":"9a03bebb1d99c5903de01f85e85ce210069506df3c6caf35c856d04793e8cf6b"}
Sep 04 22:17:43.392490 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:43.392259    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"
Sep 04 22:17:43.415402 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:43.415340    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd" podStartSLOduration=2.089425054 podStartE2EDuration="3.415320481s" totalImagesPullingTime="1.325895427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:17:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:17:43.413396865 +0000 UTC m=+546.932119400" watchObservedRunningTime="2026-09-04 22:17:43.415320481 +0000 UTC m=+546.934043030"
Sep 04 22:17:44.398437 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:44.398395    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc" podUID="5d486697-32ca-4b81-ae92-513de62a203c" containerName="manager" containerID="cri-o://7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c" gracePeriod=2
Sep 04 22:17:44.633900 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:44.633873    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:44.636344 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:17:44.636300    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-cqdrc\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="5d486697-32ca-4b81-ae92-513de62a203c" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:44.663862 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:44.663838    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9t8qn\" (UniqueName: \"kubernetes.io/projected/5d486697-32ca-4b81-ae92-513de62a203c-kube-api-access-9t8qn\") pod \"5d486697-32ca-4b81-ae92-513de62a203c\" (UID: \"5d486697-32ca-4b81-ae92-513de62a203c\") "
Sep 04 22:17:44.663930 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:44.663871    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d486697-32ca-4b81-ae92-513de62a203c-extensions-socket-volume\") pod \"5d486697-32ca-4b81-ae92-513de62a203c\" (UID: \"5d486697-32ca-4b81-ae92-513de62a203c\") "
Sep 04 22:17:44.664179 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:44.664153    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d486697-32ca-4b81-ae92-513de62a203c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5d486697-32ca-4b81-ae92-513de62a203c" (UID: "5d486697-32ca-4b81-ae92-513de62a203c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:17:44.666029 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:44.666004    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d486697-32ca-4b81-ae92-513de62a203c-kube-api-access-9t8qn" (OuterVolumeSpecName: "kube-api-access-9t8qn") pod "5d486697-32ca-4b81-ae92-513de62a203c" (UID: "5d486697-32ca-4b81-ae92-513de62a203c"). InnerVolumeSpecName "kube-api-access-9t8qn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:17:44.765213 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:44.765178    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t8qn\" (UniqueName: \"kubernetes.io/projected/5d486697-32ca-4b81-ae92-513de62a203c-kube-api-access-9t8qn\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:17:44.765213 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:44.765208    2576 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5d486697-32ca-4b81-ae92-513de62a203c-extensions-socket-volume\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:17:44.917704 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:44.917624    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d486697-32ca-4b81-ae92-513de62a203c" path="/var/lib/kubelet/pods/5d486697-32ca-4b81-ae92-513de62a203c/volumes"
Sep 04 22:17:45.404933 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:45.404902    2576 generic.go:356] "Generic (PLEG): container finished" podID="5d486697-32ca-4b81-ae92-513de62a203c" containerID="7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c" exitCode=2
Sep 04 22:17:45.405430 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:45.404950    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:45.405430 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:45.404972    2576 scope.go:117] "RemoveContainer" containerID="7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c"
Sep 04 22:17:45.409489 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:17:45.409463    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-cqdrc\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="5d486697-32ca-4b81-ae92-513de62a203c" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:45.413280 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:45.413264    2576 scope.go:117] "RemoveContainer" containerID="7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c"
Sep 04 22:17:45.413540 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:17:45.413524    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c\": container with ID starting with 7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c not found: ID does not exist" containerID="7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c"
Sep 04 22:17:45.413583 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:45.413547    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c"} err="failed to get container status \"7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c\": rpc error: code = NotFound desc = could not find container \"7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c\": container with ID starting with 7cb5c62bc344503e3d1a7a561b009afe942222026350a9296ff6db2c8a135e8c not found: ID does not exist"
Sep 04 22:17:45.912335 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:45.912294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:45.971302 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:45.971268    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:46.512405 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:46.512372    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:46.613401 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:46.613351    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:46.917273 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:17:46.917185    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6bd956ccf-cqdrc\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="5d486697-32ca-4b81-ae92-513de62a203c" pod="kuadrant-system/kuadrant-operator-controller-manager-6bd956ccf-cqdrc"
Sep 04 22:17:49.914270 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:49.914237    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:50.014875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:50.014837    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:51.552196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:51.552152    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:51.648400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:51.648358    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:54.112141 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:54.112105    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:54.214998 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:54.214959    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:54.319355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:54.319321    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:54.401584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:54.401501    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-7nxmd"
Sep 04 22:17:54.420550 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:54.420517    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:54.524126 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:54.524092    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:54.623772 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:54.623731    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:59.269516 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:59.269473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:17:59.341696 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:17:59.341664    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:00.459852 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:00.459818    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:00.561173 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:00.561140    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:09.527549 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:09.527516    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:09.625633 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:09.625598    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:23.817828 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:23.817784    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:23.918799 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:23.918764    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:24.020155 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.020118    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:24.244917 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.244882    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:18:24.245278 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.245265    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d486697-32ca-4b81-ae92-513de62a203c" containerName="manager"
Sep 04 22:18:24.245328 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.245280    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d486697-32ca-4b81-ae92-513de62a203c" containerName="manager"
Sep 04 22:18:24.245364 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.245355    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d486697-32ca-4b81-ae92-513de62a203c" containerName="manager"
Sep 04 22:18:24.248526 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.248506    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:24.251892 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.251875    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 04 22:18:24.251999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.251876    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bp59r\""
Sep 04 22:18:24.260312 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.260290    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:18:24.293109 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.293079    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:18:24.318204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.318183    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed6d1105-257b-414b-897a-569f001d4e02-config-file\") pod \"limitador-limitador-85d9db4d5d-xwzdn\" (UID: \"ed6d1105-257b-414b-897a-569f001d4e02\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:24.318308 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.318217    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5pvm\" (UniqueName: \"kubernetes.io/projected/ed6d1105-257b-414b-897a-569f001d4e02-kube-api-access-x5pvm\") pod \"limitador-limitador-85d9db4d5d-xwzdn\" (UID: \"ed6d1105-257b-414b-897a-569f001d4e02\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:24.419087 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.419045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5pvm\" (UniqueName: \"kubernetes.io/projected/ed6d1105-257b-414b-897a-569f001d4e02-kube-api-access-x5pvm\") pod \"limitador-limitador-85d9db4d5d-xwzdn\" (UID: \"ed6d1105-257b-414b-897a-569f001d4e02\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:24.419201 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.419189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed6d1105-257b-414b-897a-569f001d4e02-config-file\") pod \"limitador-limitador-85d9db4d5d-xwzdn\" (UID: \"ed6d1105-257b-414b-897a-569f001d4e02\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:24.419718 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.419701    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/ed6d1105-257b-414b-897a-569f001d4e02-config-file\") pod \"limitador-limitador-85d9db4d5d-xwzdn\" (UID: \"ed6d1105-257b-414b-897a-569f001d4e02\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:24.432091 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.432065    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5pvm\" (UniqueName: \"kubernetes.io/projected/ed6d1105-257b-414b-897a-569f001d4e02-kube-api-access-x5pvm\") pod \"limitador-limitador-85d9db4d5d-xwzdn\" (UID: \"ed6d1105-257b-414b-897a-569f001d4e02\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:24.558211 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.558150    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:24.705312 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.705272    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:24.803013 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.802977    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:24.830766 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:24.830747    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:18:24.833173 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:18:24.833144    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6d1105_257b_414b_897a_569f001d4e02.slice/crio-56a5f55080dad668d64bf7dc8b0475474a5fb3eb325112752437bea9d8166b16 WatchSource:0}: Error finding container 56a5f55080dad668d64bf7dc8b0475474a5fb3eb325112752437bea9d8166b16: Status 404 returned error can't find the container with id 56a5f55080dad668d64bf7dc8b0475474a5fb3eb325112752437bea9d8166b16
Sep 04 22:18:25.606430 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:25.606396    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn" event={"ID":"ed6d1105-257b-414b-897a-569f001d4e02","Type":"ContainerStarted","Data":"56a5f55080dad668d64bf7dc8b0475474a5fb3eb325112752437bea9d8166b16"}
Sep 04 22:18:26.612339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:26.612249    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn" event={"ID":"ed6d1105-257b-414b-897a-569f001d4e02","Type":"ContainerStarted","Data":"6a65803c57aac6285c82f4c2b6e15f8fd2bf545ac5a101ee93ddc5119e3fa45e"}
Sep 04 22:18:26.612749 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:26.612384    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:26.629299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:26.629249    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn" podStartSLOduration=1.109525826 podStartE2EDuration="2.62923205s" totalImagesPullingTime="1.519706224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:18:26.628257594 +0000 UTC m=+590.146980128" watchObservedRunningTime="2026-09-04 22:18:26.62923205 +0000 UTC m=+590.147954620"
Sep 04 22:18:29.137947 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:29.137912    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:36.864764 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:36.864732    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:18:36.865356 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:36.864961    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:18:37.622024 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:37.621994    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"
Sep 04 22:18:43.557446 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:43.557399    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:43.664756 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:43.664717    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:44.103547 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:44.103513    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:56.613407 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:56.613370    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:56.728481 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:56.728439    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:18:57.112365 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.112325    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-bd59d668-bhwt4"]
Sep 04 22:18:57.115848 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.115822    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:18:57.119477 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.119453    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 04 22:18:57.119601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.119453    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hl4x7\""
Sep 04 22:18:57.123556 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.123534    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd59d668-bhwt4"]
Sep 04 22:18:57.229194 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.229154    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdq29\" (UniqueName: \"kubernetes.io/projected/79282852-80dc-4690-8013-ea1f0e68ca40-kube-api-access-cdq29\") pod \"authorino-bd59d668-bhwt4\" (UID: \"79282852-80dc-4690-8013-ea1f0e68ca40\") " pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:18:57.229375 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.229215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/79282852-80dc-4690-8013-ea1f0e68ca40-tls-cert\") pod \"authorino-bd59d668-bhwt4\" (UID: \"79282852-80dc-4690-8013-ea1f0e68ca40\") " pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:18:57.330445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.330409    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdq29\" (UniqueName: \"kubernetes.io/projected/79282852-80dc-4690-8013-ea1f0e68ca40-kube-api-access-cdq29\") pod \"authorino-bd59d668-bhwt4\" (UID: \"79282852-80dc-4690-8013-ea1f0e68ca40\") " pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:18:57.330643 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.330464    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/79282852-80dc-4690-8013-ea1f0e68ca40-tls-cert\") pod \"authorino-bd59d668-bhwt4\" (UID: \"79282852-80dc-4690-8013-ea1f0e68ca40\") " pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:18:57.332787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.332759    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/79282852-80dc-4690-8013-ea1f0e68ca40-tls-cert\") pod \"authorino-bd59d668-bhwt4\" (UID: \"79282852-80dc-4690-8013-ea1f0e68ca40\") " pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:18:57.339044 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.339013    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdq29\" (UniqueName: \"kubernetes.io/projected/79282852-80dc-4690-8013-ea1f0e68ca40-kube-api-access-cdq29\") pod \"authorino-bd59d668-bhwt4\" (UID: \"79282852-80dc-4690-8013-ea1f0e68ca40\") " pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:18:57.434799 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.434709    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:18:57.705784 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.705729    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-bd59d668-bhwt4"]
Sep 04 22:18:57.708126 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:18:57.708085    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79282852_80dc_4690_8013_ea1f0e68ca40.slice/crio-19a87782efe5e17604a0865c428ab490a428921a8091be9ec74ff09c7de4d222 WatchSource:0}: Error finding container 19a87782efe5e17604a0865c428ab490a428921a8091be9ec74ff09c7de4d222: Status 404 returned error can't find the container with id 19a87782efe5e17604a0865c428ab490a428921a8091be9ec74ff09c7de4d222
Sep 04 22:18:57.760471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:57.760437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd59d668-bhwt4" event={"ID":"79282852-80dc-4690-8013-ea1f0e68ca40","Type":"ContainerStarted","Data":"19a87782efe5e17604a0865c428ab490a428921a8091be9ec74ff09c7de4d222"}
Sep 04 22:18:59.774013 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:59.773962    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd59d668-bhwt4" event={"ID":"79282852-80dc-4690-8013-ea1f0e68ca40","Type":"ContainerStarted","Data":"cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc"}
Sep 04 22:18:59.791532 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:18:59.791467    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-bd59d668-bhwt4" podStartSLOduration=0.869082943 podStartE2EDuration="2.791449564s" totalImagesPullingTime="1.922366621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:18:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:18:59.791224126 +0000 UTC m=+623.309946660" watchObservedRunningTime="2026-09-04 22:18:59.791449564 +0000 UTC m=+623.310172098"
Sep 04 22:19:04.919543 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:04.919504    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:04.998457 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:04.998414    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:05.060822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:05.060778    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:05.126822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:05.126781    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:05.426448 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:05.426406    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:08.697581 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:08.697543    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:09.296477 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:09.296439    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:40.673183 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:40.673144    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:40.764232 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:40.764191    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:42.228292 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:42.228236    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:42.330846 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:42.330799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:42.418948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:42.418907    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:42.526978 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:42.526893    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:42.770973 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:42.770940    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:42.872325 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:42.872223    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:42.893790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:42.893754    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:19:52.618961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:52.618925    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:19:54.207500 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:19:54.207458    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:20:06.033843 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:20:06.033798    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:20:06.519433 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:20:06.519388    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:20:18.628068 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:20:18.628020    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:20:19.392878 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:20:19.392841    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:20:30.744650 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:20:30.744566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:20:31.703910 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:20:31.703872    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:20:49.822497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:20:49.822460    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:20:51.000455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:20:51.000419    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:20:59.911092 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:20:59.911042    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:21:01.496609 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:01.496574    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:21:12.195396 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:12.195355    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:21:18.331321 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:18.331272    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:21:20.305009 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:20.304971    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:21:22.231853 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:22.231813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:21:23.098289 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.098257    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"]
Sep 04 22:21:23.102237 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.102217    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.105957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.105927    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 04 22:21:23.106443 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.106419    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tsxvl\""
Sep 04 22:21:23.106547 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.106529    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 04 22:21:23.106620 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.106539    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 04 22:21:23.114720 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.114693    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"]
Sep 04 22:21:23.121883 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.121856    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:21:23.210299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.210260    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.210299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.210302    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2907eae1-38ec-42f0-a8a9-bd1edc73436d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.210539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.210329    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.210539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.210444    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.210539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.210501    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.210676 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.210562    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzdd\" (UniqueName: \"kubernetes.io/projected/2907eae1-38ec-42f0-a8a9-bd1edc73436d-kube-api-access-srzdd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.311026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.310987    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.311529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.311075    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.311529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.311125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.311529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.311161    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzdd\" (UniqueName: \"kubernetes.io/projected/2907eae1-38ec-42f0-a8a9-bd1edc73436d-kube-api-access-srzdd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.311529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.311192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.311529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.311207    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2907eae1-38ec-42f0-a8a9-bd1edc73436d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.311727 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.311529    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.311727 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.311558    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.311727 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.311607    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.313375 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.313342    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2907eae1-38ec-42f0-a8a9-bd1edc73436d-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.313561 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.313541    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2907eae1-38ec-42f0-a8a9-bd1edc73436d-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.321390 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.321366    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzdd\" (UniqueName: \"kubernetes.io/projected/2907eae1-38ec-42f0-a8a9-bd1edc73436d-kube-api-access-srzdd\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-26mrm\" (UID: \"2907eae1-38ec-42f0-a8a9-bd1edc73436d\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.417501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.417411    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:23.688667 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.688640    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"]
Sep 04 22:21:23.690826 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:21:23.690796    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2907eae1_38ec_42f0_a8a9_bd1edc73436d.slice/crio-56ac5b445be59e7b578dac397c50ef61897e0e0f665581fcf86643ee15aef371 WatchSource:0}: Error finding container 56ac5b445be59e7b578dac397c50ef61897e0e0f665581fcf86643ee15aef371: Status 404 returned error can't find the container with id 56ac5b445be59e7b578dac397c50ef61897e0e0f665581fcf86643ee15aef371
Sep 04 22:21:23.692344 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:23.692326    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 22:21:24.470392 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:24.470349    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm" event={"ID":"2907eae1-38ec-42f0-a8a9-bd1edc73436d","Type":"ContainerStarted","Data":"56ac5b445be59e7b578dac397c50ef61897e0e0f665581fcf86643ee15aef371"}
Sep 04 22:21:25.476885 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:25.476846    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm" event={"ID":"2907eae1-38ec-42f0-a8a9-bd1edc73436d","Type":"ContainerStarted","Data":"6242a8cea2ff3713915aa8b2c11c67e2c2ca80f7db8a82c39f514204dd3098ce"}
Sep 04 22:21:25.497922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:25.497873    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm" podStartSLOduration=1.464069513 podStartE2EDuration="2.497855483s" totalImagesPullingTime="1.03378597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:21:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:21:25.496229667 +0000 UTC m=+769.014952212" watchObservedRunningTime="2026-09-04 22:21:25.497855483 +0000 UTC m=+769.016578015"
Sep 04 22:21:26.402361 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:26.402323    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:21:33.417555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:33.417514    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:33.417969 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:33.417569    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:33.430076 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:33.430031    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:33.528177 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:33.528147    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-26mrm"
Sep 04 22:21:36.291084 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:36.291023    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:21:36.356479 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:21:36.356440    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:24.265086 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.265037    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6b947f4665-x7l5d"]
Sep 04 22:22:24.268852 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.268834    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b947f4665-x7l5d"
Sep 04 22:22:24.281364 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.281330    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b947f4665-x7l5d"]
Sep 04 22:22:24.381461 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.381427    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsz7k\" (UniqueName: \"kubernetes.io/projected/3208418d-587d-47c8-889a-1b32a68c57d5-kube-api-access-vsz7k\") pod \"authorino-6b947f4665-x7l5d\" (UID: \"3208418d-587d-47c8-889a-1b32a68c57d5\") " pod="kuadrant-system/authorino-6b947f4665-x7l5d"
Sep 04 22:22:24.381657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.381480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3208418d-587d-47c8-889a-1b32a68c57d5-tls-cert\") pod \"authorino-6b947f4665-x7l5d\" (UID: \"3208418d-587d-47c8-889a-1b32a68c57d5\") " pod="kuadrant-system/authorino-6b947f4665-x7l5d"
Sep 04 22:22:24.482597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.482561    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3208418d-587d-47c8-889a-1b32a68c57d5-tls-cert\") pod \"authorino-6b947f4665-x7l5d\" (UID: \"3208418d-587d-47c8-889a-1b32a68c57d5\") " pod="kuadrant-system/authorino-6b947f4665-x7l5d"
Sep 04 22:22:24.482797 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.482662    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsz7k\" (UniqueName: \"kubernetes.io/projected/3208418d-587d-47c8-889a-1b32a68c57d5-kube-api-access-vsz7k\") pod \"authorino-6b947f4665-x7l5d\" (UID: \"3208418d-587d-47c8-889a-1b32a68c57d5\") " pod="kuadrant-system/authorino-6b947f4665-x7l5d"
Sep 04 22:22:24.485125 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.485091    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3208418d-587d-47c8-889a-1b32a68c57d5-tls-cert\") pod \"authorino-6b947f4665-x7l5d\" (UID: \"3208418d-587d-47c8-889a-1b32a68c57d5\") " pod="kuadrant-system/authorino-6b947f4665-x7l5d"
Sep 04 22:22:24.491592 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.491572    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsz7k\" (UniqueName: \"kubernetes.io/projected/3208418d-587d-47c8-889a-1b32a68c57d5-kube-api-access-vsz7k\") pod \"authorino-6b947f4665-x7l5d\" (UID: \"3208418d-587d-47c8-889a-1b32a68c57d5\") " pod="kuadrant-system/authorino-6b947f4665-x7l5d"
Sep 04 22:22:24.577786 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.577699    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6b947f4665-x7l5d"
Sep 04 22:22:24.849202 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:24.849178    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6b947f4665-x7l5d"]
Sep 04 22:22:24.851692 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:22:24.851660    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3208418d_587d_47c8_889a_1b32a68c57d5.slice/crio-2d0e350641ec39729ab227ce8477e6a9ce3581e19cc752ea62fc045a1c5b4aa4 WatchSource:0}: Error finding container 2d0e350641ec39729ab227ce8477e6a9ce3581e19cc752ea62fc045a1c5b4aa4: Status 404 returned error can't find the container with id 2d0e350641ec39729ab227ce8477e6a9ce3581e19cc752ea62fc045a1c5b4aa4
Sep 04 22:22:25.773140 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:25.773099    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b947f4665-x7l5d" event={"ID":"3208418d-587d-47c8-889a-1b32a68c57d5","Type":"ContainerStarted","Data":"4165a658238641c8b6c2a10e077791b72fa83fc88ab841b47cf25c3ec527bd15"}
Sep 04 22:22:25.773649 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:25.773154    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6b947f4665-x7l5d" event={"ID":"3208418d-587d-47c8-889a-1b32a68c57d5","Type":"ContainerStarted","Data":"2d0e350641ec39729ab227ce8477e6a9ce3581e19cc752ea62fc045a1c5b4aa4"}
Sep 04 22:22:25.793663 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:25.793608    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6b947f4665-x7l5d" podStartSLOduration=1.122502023 podStartE2EDuration="1.793590132s" totalImagesPullingTime="671.088109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:22:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:22:25.790911047 +0000 UTC m=+829.309633582" watchObservedRunningTime="2026-09-04 22:22:25.793590132 +0000 UTC m=+829.312312665"
Sep 04 22:22:25.823269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:25.823233    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd59d668-bhwt4"]
Sep 04 22:22:25.823481 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:25.823453    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-bd59d668-bhwt4" podUID="79282852-80dc-4690-8013-ea1f0e68ca40" containerName="authorino" containerID="cri-o://cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc" gracePeriod=30
Sep 04 22:22:26.077022 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.076982    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:22:26.198845 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.198812    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/79282852-80dc-4690-8013-ea1f0e68ca40-tls-cert\") pod \"79282852-80dc-4690-8013-ea1f0e68ca40\" (UID: \"79282852-80dc-4690-8013-ea1f0e68ca40\") "
Sep 04 22:22:26.199030 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.198902    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdq29\" (UniqueName: \"kubernetes.io/projected/79282852-80dc-4690-8013-ea1f0e68ca40-kube-api-access-cdq29\") pod \"79282852-80dc-4690-8013-ea1f0e68ca40\" (UID: \"79282852-80dc-4690-8013-ea1f0e68ca40\") "
Sep 04 22:22:26.201514 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.201479    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79282852-80dc-4690-8013-ea1f0e68ca40-kube-api-access-cdq29" (OuterVolumeSpecName: "kube-api-access-cdq29") pod "79282852-80dc-4690-8013-ea1f0e68ca40" (UID: "79282852-80dc-4690-8013-ea1f0e68ca40"). InnerVolumeSpecName "kube-api-access-cdq29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:22:26.209747 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.209711    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79282852-80dc-4690-8013-ea1f0e68ca40-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "79282852-80dc-4690-8013-ea1f0e68ca40" (UID: "79282852-80dc-4690-8013-ea1f0e68ca40"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:22:26.300787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.300746    2576 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/79282852-80dc-4690-8013-ea1f0e68ca40-tls-cert\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:22:26.300787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.300778    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdq29\" (UniqueName: \"kubernetes.io/projected/79282852-80dc-4690-8013-ea1f0e68ca40-kube-api-access-cdq29\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:22:26.372229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.372134    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:26.474293 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.474256    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:26.779635 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.779602    2576 generic.go:356] "Generic (PLEG): container finished" podID="79282852-80dc-4690-8013-ea1f0e68ca40" containerID="cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc" exitCode=0
Sep 04 22:22:26.780169 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.779649    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-bd59d668-bhwt4"
Sep 04 22:22:26.780169 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.779699    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd59d668-bhwt4" event={"ID":"79282852-80dc-4690-8013-ea1f0e68ca40","Type":"ContainerDied","Data":"cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc"}
Sep 04 22:22:26.780169 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.779752    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-bd59d668-bhwt4" event={"ID":"79282852-80dc-4690-8013-ea1f0e68ca40","Type":"ContainerDied","Data":"19a87782efe5e17604a0865c428ab490a428921a8091be9ec74ff09c7de4d222"}
Sep 04 22:22:26.780169 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.779778    2576 scope.go:117] "RemoveContainer" containerID="cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc"
Sep 04 22:22:26.788720 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.788703    2576 scope.go:117] "RemoveContainer" containerID="cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc"
Sep 04 22:22:26.789035 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:22:26.789016    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc\": container with ID starting with cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc not found: ID does not exist" containerID="cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc"
Sep 04 22:22:26.789124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.789045    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc"} err="failed to get container status \"cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc\": rpc error: code = NotFound desc = could not find container \"cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc\": container with ID starting with cf3eb5485f418501812730857c6133caaada3767ba4f0839a6afce9dd26c7cbc not found: ID does not exist"
Sep 04 22:22:26.802462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.802435    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-bd59d668-bhwt4"]
Sep 04 22:22:26.804880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.804854    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-bd59d668-bhwt4"]
Sep 04 22:22:26.917989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:26.917950    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79282852-80dc-4690-8013-ea1f0e68ca40" path="/var/lib/kubelet/pods/79282852-80dc-4690-8013-ea1f0e68ca40/volumes"
Sep 04 22:22:27.506186 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:27.506141    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:27.603822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:27.603784    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:43.239134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:43.239100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:43.353428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:43.353394    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:43.603494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:43.603400    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:43.735459 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:43.735425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:43.835447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:43.835415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:44.627081 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:44.627025    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:44.723216 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:44.723177    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:45.205729 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.205691    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7752-a92a52"]
Sep 04 22:22:45.206099 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.206085    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79282852-80dc-4690-8013-ea1f0e68ca40" containerName="authorino"
Sep 04 22:22:45.206155 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.206100    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="79282852-80dc-4690-8013-ea1f0e68ca40" containerName="authorino"
Sep 04 22:22:45.206191 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.206181    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="79282852-80dc-4690-8013-ea1f0e68ca40" containerName="authorino"
Sep 04 22:22:45.210611 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.210594    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-a92a52"
Sep 04 22:22:45.214067 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.214036    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dlrbm\""
Sep 04 22:22:45.221272 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.221249    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-a92a52"]
Sep 04 22:22:45.351555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.351523    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:45.381708 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.381679    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qcm\" (UniqueName: \"kubernetes.io/projected/0c450dbd-4005-4d03-aed6-c829f0761fcf-kube-api-access-t2qcm\") pod \"test-curl-7752-a92a52\" (UID: \"0c450dbd-4005-4d03-aed6-c829f0761fcf\") " pod="opendatahub/test-curl-7752-a92a52"
Sep 04 22:22:45.482493 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.482396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qcm\" (UniqueName: \"kubernetes.io/projected/0c450dbd-4005-4d03-aed6-c829f0761fcf-kube-api-access-t2qcm\") pod \"test-curl-7752-a92a52\" (UID: \"0c450dbd-4005-4d03-aed6-c829f0761fcf\") " pod="opendatahub/test-curl-7752-a92a52"
Sep 04 22:22:45.493286 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.493254    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qcm\" (UniqueName: \"kubernetes.io/projected/0c450dbd-4005-4d03-aed6-c829f0761fcf-kube-api-access-t2qcm\") pod \"test-curl-7752-a92a52\" (UID: \"0c450dbd-4005-4d03-aed6-c829f0761fcf\") " pod="opendatahub/test-curl-7752-a92a52"
Sep 04 22:22:45.520208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.520181    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-a92a52"
Sep 04 22:22:45.795114 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.795086    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-a92a52"]
Sep 04 22:22:45.797982 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:22:45.797949    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c450dbd_4005_4d03_aed6_c829f0761fcf.slice/crio-d82191c2ef21540ab33ebdcfa073bd4f58864c7bf5166570135c18128926e950 WatchSource:0}: Error finding container d82191c2ef21540ab33ebdcfa073bd4f58864c7bf5166570135c18128926e950: Status 404 returned error can't find the container with id d82191c2ef21540ab33ebdcfa073bd4f58864c7bf5166570135c18128926e950
Sep 04 22:22:45.870468 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:45.870432    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-a92a52" event={"ID":"0c450dbd-4005-4d03-aed6-c829f0761fcf","Type":"ContainerStarted","Data":"d82191c2ef21540ab33ebdcfa073bd4f58864c7bf5166570135c18128926e950"}
Sep 04 22:22:46.585438 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:46.585401    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:46.682773 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:46.682722    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:47.248526 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:47.248488    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:22:47.866386 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:47.866297    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:47.963572 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:47.963538    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:48.091214 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:48.091119    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:48.504083 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:48.504034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:48.606130 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:48.606077    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:48.891609 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:48.891518    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-a92a52" event={"ID":"0c450dbd-4005-4d03-aed6-c829f0761fcf","Type":"ContainerStarted","Data":"fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50"}
Sep 04 22:22:48.907903 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:48.907856    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7752-a92a52" podStartSLOduration=1.877744366 podStartE2EDuration="3.907839272s" totalImagesPullingTime="2.030094906s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:22:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:22:48.907586497 +0000 UTC m=+852.426309032" watchObservedRunningTime="2026-09-04 22:22:48.907839272 +0000 UTC m=+852.426561818"
Sep 04 22:22:49.483722 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.483685    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7752-a92a52"]
Sep 04 22:22:49.488645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.488617    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7752-a92a52"]
Sep 04 22:22:49.706464 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.706431    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7752-751f7e"]
Sep 04 22:22:49.706833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.706811    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c450dbd-4005-4d03-aed6-c829f0761fcf" containerName="test-curl-7752-a92a52"
Sep 04 22:22:49.706833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.706824    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c450dbd-4005-4d03-aed6-c829f0761fcf" containerName="test-curl-7752-a92a52"
Sep 04 22:22:49.706921 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.706909    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c450dbd-4005-4d03-aed6-c829f0761fcf" containerName="test-curl-7752-a92a52"
Sep 04 22:22:49.710209 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.710192    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-751f7e"
Sep 04 22:22:49.714621 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.714595    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-751f7e"]
Sep 04 22:22:49.721690 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.721668    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlkcj\" (UniqueName: \"kubernetes.io/projected/8c2bb568-9645-4a69-9959-6041877f41aa-kube-api-access-vlkcj\") pod \"test-curl-7752-751f7e\" (UID: \"8c2bb568-9645-4a69-9959-6041877f41aa\") " pod="opendatahub/test-curl-7752-751f7e"
Sep 04 22:22:49.822914 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.822875    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlkcj\" (UniqueName: \"kubernetes.io/projected/8c2bb568-9645-4a69-9959-6041877f41aa-kube-api-access-vlkcj\") pod \"test-curl-7752-751f7e\" (UID: \"8c2bb568-9645-4a69-9959-6041877f41aa\") " pod="opendatahub/test-curl-7752-751f7e"
Sep 04 22:22:49.831513 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:49.831454    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlkcj\" (UniqueName: \"kubernetes.io/projected/8c2bb568-9645-4a69-9959-6041877f41aa-kube-api-access-vlkcj\") pod \"test-curl-7752-751f7e\" (UID: \"8c2bb568-9645-4a69-9959-6041877f41aa\") " pod="opendatahub/test-curl-7752-751f7e"
Sep 04 22:22:49.896592 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:22:49.896561    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7752-a92a52" container="test-curl-7752-a92a52"
Sep 04 22:22:50.020614 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:50.020534    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-751f7e"
Sep 04 22:22:50.291464 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:50.291440    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7752-751f7e"]
Sep 04 22:22:50.293827 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:22:50.293801    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c2bb568_9645_4a69_9959_6041877f41aa.slice/crio-f2ff6adcad6a689e9080286bb0370c3505194691da6ec5c62bf297e1e632908f WatchSource:0}: Error finding container f2ff6adcad6a689e9080286bb0370c3505194691da6ec5c62bf297e1e632908f: Status 404 returned error can't find the container with id f2ff6adcad6a689e9080286bb0370c3505194691da6ec5c62bf297e1e632908f
Sep 04 22:22:50.902879 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:50.902837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-751f7e" event={"ID":"8c2bb568-9645-4a69-9959-6041877f41aa","Type":"ContainerStarted","Data":"8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143"}
Sep 04 22:22:50.902879 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:50.902881    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7752-751f7e" event={"ID":"8c2bb568-9645-4a69-9959-6041877f41aa","Type":"ContainerStarted","Data":"f2ff6adcad6a689e9080286bb0370c3505194691da6ec5c62bf297e1e632908f"}
Sep 04 22:22:50.903424 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:50.902950    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7752-a92a52" podUID="0c450dbd-4005-4d03-aed6-c829f0761fcf" containerName="test-curl-7752-a92a52" containerID="cri-o://fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50" gracePeriod=2
Sep 04 22:22:50.919958 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:50.919910    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7752-751f7e" podStartSLOduration=1.919896546 podStartE2EDuration="1.919896546s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:22:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:22:50.917906798 +0000 UTC m=+854.436629332" watchObservedRunningTime="2026-09-04 22:22:50.919896546 +0000 UTC m=+854.438619079"
Sep 04 22:22:51.529295 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:51.529262    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7752-751f7e"]
Sep 04 22:22:51.535066 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:51.535020    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7752-751f7e"]
Sep 04 22:22:51.826073 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:51.826020    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:51.908284 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:22:51.908254    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7752-751f7e" container="test-curl-7752-751f7e"
Sep 04 22:22:51.927163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:51.927132    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:52.043304 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.043266    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:52.141592 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.141498    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:52.337734 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.337700    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"]
Sep 04 22:22:52.338299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.338280    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c2bb568-9645-4a69-9959-6041877f41aa" containerName="test-curl-7752-751f7e"
Sep 04 22:22:52.338299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.338299    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c2bb568-9645-4a69-9959-6041877f41aa" containerName="test-curl-7752-751f7e"
Sep 04 22:22:52.338437 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.338382    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c2bb568-9645-4a69-9959-6041877f41aa" containerName="test-curl-7752-751f7e"
Sep 04 22:22:52.341547 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.341529    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.344238 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.344215    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-a8d845-openshift-default-dockercfg-ppqdj\""
Sep 04 22:22:52.344364 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.344339    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 04 22:22:52.370307 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.370278    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"]
Sep 04 22:22:52.397377 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.397286    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:52.450173 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.450139    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-credential-socket\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.450354 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.450182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14438d60-468e-49e2-a864-4e863cc84594-istiod-ca-cert\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.450354 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.450202    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-envoy\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.450354 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.450223    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14438d60-468e-49e2-a864-4e863cc84594-istio-podinfo\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.450354 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.450256    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-istio-token\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.450354 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.450291    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-certs\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.450618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.450382    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-socket\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.450618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.450417    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsg6j\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-kube-api-access-zsg6j\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.450618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.450446    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-data\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.485226 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.485188    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:52.551115 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-credential-socket\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551313 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551123    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14438d60-468e-49e2-a864-4e863cc84594-istiod-ca-cert\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551313 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551146    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-envoy\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551313 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551164    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14438d60-468e-49e2-a864-4e863cc84594-istio-podinfo\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551313 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551194    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-istio-token\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551313 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551237    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-certs\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551313 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551285    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-socket\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551316    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsg6j\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-kube-api-access-zsg6j\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-data\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551546    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-credential-socket\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.551921 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.551893    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-data\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.552086 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.552002    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-certs\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.552151 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.552102    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-socket\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.552213 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.552142    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14438d60-468e-49e2-a864-4e863cc84594-istiod-ca-cert\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.553646 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.553622    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-envoy\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.553822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.553801    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14438d60-468e-49e2-a864-4e863cc84594-istio-podinfo\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.560257 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.560228    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-istio-token\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.560352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.560295    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsg6j\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-kube-api-access-zsg6j\") pod \"e2e-autores-a8d845-openshift-default-8696b856d6-b96tb\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") " pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.618474 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.618436    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:52.652600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.652515    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:52.914284 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.914244    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7752-751f7e" podUID="8c2bb568-9645-4a69-9959-6041877f41aa" containerName="test-curl-7752-751f7e" containerID="cri-o://8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143" gracePeriod=2
Sep 04 22:22:52.917156 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:22:52.917100    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7752-751f7e\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="8c2bb568-9645-4a69-9959-6041877f41aa" pod="opendatahub/test-curl-7752-751f7e"
Sep 04 22:22:52.970954 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:52.970930    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"]
Sep 04 22:22:52.981478 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:22:52.981439    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14438d60_468e_49e2_a864_4e863cc84594.slice/crio-0f7d4744533491348ef885f2fb473f930d6c03c7adbcbd13d21df69c4add05f1 WatchSource:0}: Error finding container 0f7d4744533491348ef885f2fb473f930d6c03c7adbcbd13d21df69c4add05f1: Status 404 returned error can't find the container with id 0f7d4744533491348ef885f2fb473f930d6c03c7adbcbd13d21df69c4add05f1
Sep 04 22:22:53.076164 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.076135    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-a92a52"
Sep 04 22:22:53.157181 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.157091    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2qcm\" (UniqueName: \"kubernetes.io/projected/0c450dbd-4005-4d03-aed6-c829f0761fcf-kube-api-access-t2qcm\") pod \"0c450dbd-4005-4d03-aed6-c829f0761fcf\" (UID: \"0c450dbd-4005-4d03-aed6-c829f0761fcf\") "
Sep 04 22:22:53.159300 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.159272    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c450dbd-4005-4d03-aed6-c829f0761fcf-kube-api-access-t2qcm" (OuterVolumeSpecName: "kube-api-access-t2qcm") pod "0c450dbd-4005-4d03-aed6-c829f0761fcf" (UID: "0c450dbd-4005-4d03-aed6-c829f0761fcf"). InnerVolumeSpecName "kube-api-access-t2qcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:22:53.258386 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.258349    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qcm\" (UniqueName: \"kubernetes.io/projected/0c450dbd-4005-4d03-aed6-c829f0761fcf-kube-api-access-t2qcm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:22:53.921182 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.921087    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" event={"ID":"14438d60-468e-49e2-a864-4e863cc84594","Type":"ContainerStarted","Data":"0f7d4744533491348ef885f2fb473f930d6c03c7adbcbd13d21df69c4add05f1"}
Sep 04 22:22:53.922322 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.922298    2576 generic.go:356] "Generic (PLEG): container finished" podID="0c450dbd-4005-4d03-aed6-c829f0761fcf" containerID="fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50" exitCode=137
Sep 04 22:22:53.922430 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.922337    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-a92a52"
Sep 04 22:22:53.922430 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.922377    2576 scope.go:117] "RemoveContainer" containerID="fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50"
Sep 04 22:22:53.931737 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.931714    2576 scope.go:117] "RemoveContainer" containerID="fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50"
Sep 04 22:22:53.932048 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:22:53.932022    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50\": container with ID starting with fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50 not found: ID does not exist" containerID="fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50"
Sep 04 22:22:53.932136 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:53.932077    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50"} err="failed to get container status \"fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50\": rpc error: code = NotFound desc = could not find container \"fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50\": container with ID starting with fcfd5e7814deebaf94a159266171a88393e425fdf0e3e87807f34a803ff74d50 not found: ID does not exist"
Sep 04 22:22:53.933566 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:22:53.933532    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7752-a92a52\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="0c450dbd-4005-4d03-aed6-c829f0761fcf" pod="opendatahub/test-curl-7752-a92a52"
Sep 04 22:22:54.884161 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:54.884120    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:54.923778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:54.923636    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c450dbd-4005-4d03-aed6-c829f0761fcf" path="/var/lib/kubelet/pods/0c450dbd-4005-4d03-aed6-c829f0761fcf/volumes"
Sep 04 22:22:54.979841 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:54.979805    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:55.851498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.851447    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:22:55.851609 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.851547    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:22:55.851609 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.851586    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:22:55.935035 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.935011    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-751f7e"
Sep 04 22:22:55.935658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.935636    2576 generic.go:356] "Generic (PLEG): container finished" podID="8c2bb568-9645-4a69-9959-6041877f41aa" containerID="8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143" exitCode=137
Sep 04 22:22:55.935743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.935730    2576 scope.go:117] "RemoveContainer" containerID="8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143"
Sep 04 22:22:55.943825 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.943802    2576 scope.go:117] "RemoveContainer" containerID="8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143"
Sep 04 22:22:55.944135 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:22:55.944111    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143\": container with ID starting with 8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143 not found: ID does not exist" containerID="8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143"
Sep 04 22:22:55.944218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.944144    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143"} err="failed to get container status \"8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143\": rpc error: code = NotFound desc = could not find container \"8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143\": container with ID starting with 8876187202b710ecab88acac4a7c7d6c8408202b2dc7c4e49636f5ff2a257143 not found: ID does not exist"
Sep 04 22:22:55.985218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.985182    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vlkcj\" (UniqueName: \"kubernetes.io/projected/8c2bb568-9645-4a69-9959-6041877f41aa-kube-api-access-vlkcj\") pod \"8c2bb568-9645-4a69-9959-6041877f41aa\" (UID: \"8c2bb568-9645-4a69-9959-6041877f41aa\") "
Sep 04 22:22:55.987300 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:55.987267    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c2bb568-9645-4a69-9959-6041877f41aa-kube-api-access-vlkcj" (OuterVolumeSpecName: "kube-api-access-vlkcj") pod "8c2bb568-9645-4a69-9959-6041877f41aa" (UID: "8c2bb568-9645-4a69-9959-6041877f41aa"). InnerVolumeSpecName "kube-api-access-vlkcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:22:56.086578 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:56.086488    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlkcj\" (UniqueName: \"kubernetes.io/projected/8c2bb568-9645-4a69-9959-6041877f41aa-kube-api-access-vlkcj\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:22:56.262616 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:56.262578    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:56.325500 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:56.325461    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:56.766445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:56.766401    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:56.918601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:56.918568    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c2bb568-9645-4a69-9959-6041877f41aa" path="/var/lib/kubelet/pods/8c2bb568-9645-4a69-9959-6041877f41aa/volumes"
Sep 04 22:22:56.923741 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:56.923713    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:56.943104 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:56.943074    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7752-751f7e"
Sep 04 22:22:56.945699 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:56.945665    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" event={"ID":"14438d60-468e-49e2-a864-4e863cc84594","Type":"ContainerStarted","Data":"ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5"}
Sep 04 22:22:56.972798 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:56.972745    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" podStartSLOduration=2.105401326 podStartE2EDuration="4.972729128s" totalImagesPullingTime="2.867327802s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:22:56.970695586 +0000 UTC m=+860.489418123" watchObservedRunningTime="2026-09-04 22:22:56.972729128 +0000 UTC m=+860.491451661"
Sep 04 22:22:57.022646 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.022560    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:57.125715 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.125681    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"]
Sep 04 22:22:57.129512 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.129495    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:22:57.132403 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.132373    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-c561f9fe\""
Sep 04 22:22:57.132539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.132462    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-c561f9fe-dockercfg-m7xz4\""
Sep 04 22:22:57.140739 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.140714    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"]
Sep 04 22:22:57.197749 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.197708    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b545c8c2-6132-4b7a-9ac6-854e65729e03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg\" (UID: \"b545c8c2-6132-4b7a-9ac6-854e65729e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:22:57.197937 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.197782    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swb8f\" (UniqueName: \"kubernetes.io/projected/b545c8c2-6132-4b7a-9ac6-854e65729e03-kube-api-access-swb8f\") pod \"payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg\" (UID: \"b545c8c2-6132-4b7a-9ac6-854e65729e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:22:57.298554 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.298450    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b545c8c2-6132-4b7a-9ac6-854e65729e03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg\" (UID: \"b545c8c2-6132-4b7a-9ac6-854e65729e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:22:57.298554 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.298522    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swb8f\" (UniqueName: \"kubernetes.io/projected/b545c8c2-6132-4b7a-9ac6-854e65729e03-kube-api-access-swb8f\") pod \"payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg\" (UID: \"b545c8c2-6132-4b7a-9ac6-854e65729e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:22:57.299240 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.299218    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b545c8c2-6132-4b7a-9ac6-854e65729e03-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg\" (UID: \"b545c8c2-6132-4b7a-9ac6-854e65729e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:22:57.307414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.307382    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swb8f\" (UniqueName: \"kubernetes.io/projected/b545c8c2-6132-4b7a-9ac6-854e65729e03-kube-api-access-swb8f\") pod \"payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg\" (UID: \"b545c8c2-6132-4b7a-9ac6-854e65729e03\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:22:57.441371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.441333    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:22:57.653765 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.653678    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:57.658237 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.658209    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:57.711316 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.711291    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"]
Sep 04 22:22:57.713447 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:22:57.713423    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb545c8c2_6132_4b7a_9ac6_854e65729e03.slice/crio-042ef37d5d53a0745b0c622cf9631f530cf5892039471ac9435d7b5ad66f5b31 WatchSource:0}: Error finding container 042ef37d5d53a0745b0c622cf9631f530cf5892039471ac9435d7b5ad66f5b31: Status 404 returned error can't find the container with id 042ef37d5d53a0745b0c622cf9631f530cf5892039471ac9435d7b5ad66f5b31
Sep 04 22:22:57.951433 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.951347    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg" event={"ID":"b545c8c2-6132-4b7a-9ac6-854e65729e03","Type":"ContainerStarted","Data":"042ef37d5d53a0745b0c622cf9631f530cf5892039471ac9435d7b5ad66f5b31"}
Sep 04 22:22:57.951816 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.951587    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:57.952519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:57.952494    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:22:58.984560 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:58.984524    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:22:59.084724 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:22:59.084683    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:00.194014 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.193979    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"]
Sep 04 22:23:00.198099 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.198066    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.201736 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.201709    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-a8d845-serving-cert\""
Sep 04 22:23:00.201878 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.201820    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 22:23:00.201878 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.201842    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 04 22:23:00.201983 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.201711    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 04 22:23:00.202073 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.202030    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2lmrb\""
Sep 04 22:23:00.212628 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.212603    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"]
Sep 04 22:23:00.234822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.234788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:00.306992 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.306945    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:00.329132 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.329095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j56w\" (UniqueName: \"kubernetes.io/projected/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-kube-api-access-9j56w\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.329290 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.329157    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-openshift-service-ca\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.329290 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.329236    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-metrics-certs\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.329381 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.329333    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-maas-api-tls\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.430447 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.430408    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j56w\" (UniqueName: \"kubernetes.io/projected/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-kube-api-access-9j56w\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.430626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.430462    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-openshift-service-ca\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.430626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.430490    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-metrics-certs\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.430791 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.430645    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-maas-api-tls\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.430791 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:23:00.430750    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-serving-cert: secret "maas-api-e2e-autores-a8d845-serving-cert" not found
Sep 04 22:23:00.430900 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:23:00.430828    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-maas-api-tls podName:7bf143ca-6576-4eaa-bb15-425cfb0b6a9a nodeName:}" failed. No retries permitted until 2026-09-04 22:23:00.930804011 +0000 UTC m=+864.449526532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-maas-api-tls") pod "maas-api-e2e-autores-a8d845-cf74469cd-94wbw" (UID: "7bf143ca-6576-4eaa-bb15-425cfb0b6a9a") : secret "maas-api-e2e-autores-a8d845-serving-cert" not found
Sep 04 22:23:00.431312 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.431277    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-openshift-service-ca\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.432961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.432937    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-metrics-certs\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.441082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.441040    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:23:00.445937 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.445911    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j56w\" (UniqueName: \"kubernetes.io/projected/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-kube-api-access-9j56w\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.634790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.634705    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:00.895099 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.894991    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:00.935222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.935183    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-maas-api-tls\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.937704 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.937679    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-maas-api-tls\") pod \"maas-api-e2e-autores-a8d845-cf74469cd-94wbw\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:00.974940 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.974901    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg" event={"ID":"b545c8c2-6132-4b7a-9ac6-854e65729e03","Type":"ContainerStarted","Data":"e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83"}
Sep 04 22:23:00.975159 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.974985    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:23:00.990598 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.990565    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:00.993519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:00.993475    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg" podStartSLOduration=1.7547861390000001 podStartE2EDuration="3.993460171s" totalImagesPullingTime="2.238674032s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:22:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:00.992195594 +0000 UTC m=+864.510918135" watchObservedRunningTime="2026-09-04 22:23:00.993460171 +0000 UTC m=+864.512182695"
Sep 04 22:23:01.109235 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:01.109195    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:01.395674 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:01.395647    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"]
Sep 04 22:23:01.397681 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:23:01.397651    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf143ca_6576_4eaa_bb15_425cfb0b6a9a.slice/crio-348bcca4e4ab550a932d9cb42cbbaefa5bf22f8c5e7a04df6e2bdd261cec0b6c WatchSource:0}: Error finding container 348bcca4e4ab550a932d9cb42cbbaefa5bf22f8c5e7a04df6e2bdd261cec0b6c: Status 404 returned error can't find the container with id 348bcca4e4ab550a932d9cb42cbbaefa5bf22f8c5e7a04df6e2bdd261cec0b6c
Sep 04 22:23:01.982198 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:01.982156    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw" event={"ID":"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a","Type":"ContainerStarted","Data":"348bcca4e4ab550a932d9cb42cbbaefa5bf22f8c5e7a04df6e2bdd261cec0b6c"}
Sep 04 22:23:02.024300 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:02.024254    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:02.202562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:02.202476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:02.299644 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:02.299552    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:02.400593 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:02.400495    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:03.995458 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:03.995418    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw" event={"ID":"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a","Type":"ContainerStarted","Data":"0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48"}
Sep 04 22:23:03.995927 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:03.995479    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:04.017305 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:04.017247    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw" podStartSLOduration=1.729236288 podStartE2EDuration="4.01722878s" totalImagesPullingTime="2.287992492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:04.014608926 +0000 UTC m=+867.533331460" watchObservedRunningTime="2026-09-04 22:23:04.01722878 +0000 UTC m=+867.535951312"
Sep 04 22:23:05.057805 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:05.057762    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:06.164092 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:06.164064    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:06.264466 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:06.264406    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:06.383392 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:06.383344    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:06.633442 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:06.633409    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:10.005930 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:10.005897    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:23:10.194547 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:10.194508    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:11.985196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:11.985161    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:23:12.872618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:12.872575    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:12.981111 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:12.981048    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:13.075957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:13.075921    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:13.224457 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:13.224413    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:16.587629 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:16.587553    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:16.907409 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:16.907321    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:17.218917 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.218880    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"]
Sep 04 22:23:17.223337 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.223312    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.226473 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.226451    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fecd36-openshift-default-dockercfg-bzf5t\""
Sep 04 22:23:17.249233 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.249201    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"]
Sep 04 22:23:17.262281 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.262244    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:17.407168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.407131    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-data\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.407168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.407166    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd604e6d-a4d9-4787-b133-f925d2b84367-istiod-ca-cert\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.407396 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.407189    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-certs\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.407396 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.407243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhtbh\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-kube-api-access-rhtbh\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.407396 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.407319    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-credential-socket\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.407396 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.407364    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-token\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.407396 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.407388    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-podinfo\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.407595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.407492    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-socket\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.407595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.407539    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-envoy\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.459028 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.458978    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:17.508892 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.508794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-envoy\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.508900    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-data\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.508928    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd604e6d-a4d9-4787-b133-f925d2b84367-istiod-ca-cert\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.508957    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-certs\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.508991    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhtbh\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-kube-api-access-rhtbh\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.509018    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-credential-socket\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509326 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.509086    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-token\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509326 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.509125    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-podinfo\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509326 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.509203    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-socket\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509485 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.509337    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-data\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.509751    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd604e6d-a4d9-4787-b133-f925d2b84367-istiod-ca-cert\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.509952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.509834    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-socket\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.510096 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.509863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-credential-socket\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.510169 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.510088    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-certs\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.511368 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.511344    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-envoy\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.511909 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.511878    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-podinfo\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.518468 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.518445    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-token\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.519221 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.519195    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhtbh\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-kube-api-access-rhtbh\") pod \"e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") " pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.535005 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.534970    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:17.553998 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.553963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:17.819382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.819351    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"]
Sep 04 22:23:17.822796 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:23:17.822749    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd604e6d_a4d9_4787_b133_f925d2b84367.slice/crio-537dc16eec6404e8006932098c413dc0c527e3ee14e2b3a71b0cf88f3361566b WatchSource:0}: Error finding container 537dc16eec6404e8006932098c413dc0c527e3ee14e2b3a71b0cf88f3361566b: Status 404 returned error can't find the container with id 537dc16eec6404e8006932098c413dc0c527e3ee14e2b3a71b0cf88f3361566b
Sep 04 22:23:17.825833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.825787    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:17.825961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.825886    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:17.825961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.825931    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:17.999555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:17.999513    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:23:18.079306 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:18.079208    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" event={"ID":"dd604e6d-a4d9-4787-b133-f925d2b84367","Type":"ContainerStarted","Data":"e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145"}
Sep 04 22:23:18.079306 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:18.079288    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" event={"ID":"dd604e6d-a4d9-4787-b133-f925d2b84367","Type":"ContainerStarted","Data":"537dc16eec6404e8006932098c413dc0c527e3ee14e2b3a71b0cf88f3361566b"}
Sep 04 22:23:18.110193 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:18.110129    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" podStartSLOduration=1.110105705 podStartE2EDuration="1.110105705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:18.109088181 +0000 UTC m=+881.627810715" watchObservedRunningTime="2026-09-04 22:23:18.110105705 +0000 UTC m=+881.628828240"
Sep 04 22:23:18.536141 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:18.536097    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:18.542314 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:18.542283    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:19.083976 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:19.083947    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:19.084995 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:19.084974    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:19.670922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:19.670884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:19.771508 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:19.771471    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:20.055042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:20.055005    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:20.147817 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:20.147571    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:21.101612 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.101575    2576 generic.go:356] "Generic (PLEG): container finished" podID="14438d60-468e-49e2-a864-4e863cc84594" containerID="ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5" exitCode=0
Sep 04 22:23:21.101791 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.101645    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" event={"ID":"14438d60-468e-49e2-a864-4e863cc84594","Type":"ContainerDied","Data":"ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5"}
Sep 04 22:23:21.102191 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.102176    2576 scope.go:117] "RemoveContainer" containerID="ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5"
Sep 04 22:23:21.103306 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.103276    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:21.103371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.103343    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:21.103407 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.103376    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:21.600855 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.600822    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"]
Sep 04 22:23:21.605182 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.604559    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.607606 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.607581    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-fecd36-serving-cert\""
Sep 04 22:23:21.615679 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.615651    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"]
Sep 04 22:23:21.653273 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.653243    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vhsd\" (UniqueName: \"kubernetes.io/projected/a43077ab-2bd6-4d17-a621-253842dcecac-kube-api-access-4vhsd\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.653435 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.653338    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-metrics-certs\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.653435 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.653381    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43077ab-2bd6-4d17-a621-253842dcecac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.653515 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.653450    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-maas-api-tls\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.669660 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.669578    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:21.753898 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.753859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43077ab-2bd6-4d17-a621-253842dcecac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.754148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.753964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-maas-api-tls\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.754148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.754004    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vhsd\" (UniqueName: \"kubernetes.io/projected/a43077ab-2bd6-4d17-a621-253842dcecac-kube-api-access-4vhsd\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.754288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.754156    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-metrics-certs\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.754288 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:23:21.754174    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-serving-cert: secret "maas-api-e2e-inf-b-fecd36-serving-cert" not found
Sep 04 22:23:21.754288 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:23:21.754256    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-maas-api-tls podName:a43077ab-2bd6-4d17-a621-253842dcecac nodeName:}" failed. No retries permitted until 2026-09-04 22:23:22.254232364 +0000 UTC m=+885.772954877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-maas-api-tls") pod "maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb" (UID: "a43077ab-2bd6-4d17-a621-253842dcecac") : secret "maas-api-e2e-inf-b-fecd36-serving-cert" not found
Sep 04 22:23:21.754776 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.754748    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43077ab-2bd6-4d17-a621-253842dcecac-openshift-service-ca\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.761077 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.757393    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-metrics-certs\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:21.763410 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.762826    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:21.769705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:21.769670    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vhsd\" (UniqueName: \"kubernetes.io/projected/a43077ab-2bd6-4d17-a621-253842dcecac-kube-api-access-4vhsd\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:22.108740 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.108707    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" event={"ID":"14438d60-468e-49e2-a864-4e863cc84594","Type":"ContainerStarted","Data":"0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987"}
Sep 04 22:23:22.200745 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.200709    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:22.271094 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.266452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-maas-api-tls\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:22.271094 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.270242    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-maas-api-tls\") pod \"maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:22.369796 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.369696    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:22.520045 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.520006    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:22.653450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.653371    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:23:22.653859 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.653458    2576 patch_prober.go:28] interesting pod/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 04 22:23:22.653859 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.653527    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 04 22:23:22.726474 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.726419    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:22.815235 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:22.815209    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"]
Sep 04 22:23:22.817154 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:23:22.817127    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43077ab_2bd6_4d17_a621_253842dcecac.slice/crio-e19f5c58d8c2b30dccbf82019e19001aad97f375d8b8fc5732312650a3c5b5c4 WatchSource:0}: Error finding container e19f5c58d8c2b30dccbf82019e19001aad97f375d8b8fc5732312650a3c5b5c4: Status 404 returned error can't find the container with id e19f5c58d8c2b30dccbf82019e19001aad97f375d8b8fc5732312650a3c5b5c4
Sep 04 22:23:23.115403 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:23.115379    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb" event={"ID":"a43077ab-2bd6-4d17-a621-253842dcecac","Type":"ContainerStarted","Data":"e19f5c58d8c2b30dccbf82019e19001aad97f375d8b8fc5732312650a3c5b5c4"}
Sep 04 22:23:23.542564 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:23.542528    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:23.653304 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:23.653269    2576 patch_prober.go:28] interesting pod/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 04 22:23:23.653480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:23.653325    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 04 22:23:23.916119 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:23.916010    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:24.122567 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:24.122528    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb" event={"ID":"a43077ab-2bd6-4d17-a621-253842dcecac","Type":"ContainerStarted","Data":"3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98"}
Sep 04 22:23:24.122769 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:24.122619    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:24.547426 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:24.547383    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:24.652921 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:24.652884    2576 patch_prober.go:28] interesting pod/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 04 22:23:24.653127 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:24.652939    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 04 22:23:25.653127 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:25.653091    2576 patch_prober.go:28] interesting pod/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 04 22:23:25.653523 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:25.653148    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 04 22:23:26.653245 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:26.653191    2576 patch_prober.go:28] interesting pod/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Sep 04 22:23:26.653664 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:26.653264    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Sep 04 22:23:27.657410 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:27.657372    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:23:27.657781 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:27.657642    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:23:27.658342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:27.658324    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:23:27.679681 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:27.679633    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb" podStartSLOduration=6.388782465 podStartE2EDuration="6.679619091s" totalImagesPullingTime="290.836626ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:24.169249428 +0000 UTC m=+887.687971965" watchObservedRunningTime="2026-09-04 22:23:27.679619091 +0000 UTC m=+891.198341643"
Sep 04 22:23:30.134795 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:30.134708    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:23:32.402263 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:32.402223    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:23:36.328534 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:36.328497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:36.911474 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:36.911444    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:23:36.912638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:36.912613    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:23:37.086689 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.086628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:37.125957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.125921    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"]
Sep 04 22:23:37.126212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.126164    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg" podUID="b545c8c2-6132-4b7a-9ac6-854e65729e03" containerName="payload-pre-processing" containerID="cri-o://e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83" gracePeriod=30
Sep 04 22:23:37.178142 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.177838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:37.277079 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.269524    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:37.376080 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.375597    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:37.414012 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.413986    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:23:37.482040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.481998    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:37.533762 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.533727    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swb8f\" (UniqueName: \"kubernetes.io/projected/b545c8c2-6132-4b7a-9ac6-854e65729e03-kube-api-access-swb8f\") pod \"b545c8c2-6132-4b7a-9ac6-854e65729e03\" (UID: \"b545c8c2-6132-4b7a-9ac6-854e65729e03\") "
Sep 04 22:23:37.533958 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.533811    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b545c8c2-6132-4b7a-9ac6-854e65729e03-plugins-config-volume\") pod \"b545c8c2-6132-4b7a-9ac6-854e65729e03\" (UID: \"b545c8c2-6132-4b7a-9ac6-854e65729e03\") "
Sep 04 22:23:37.534187 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.534156    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b545c8c2-6132-4b7a-9ac6-854e65729e03-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b545c8c2-6132-4b7a-9ac6-854e65729e03" (UID: "b545c8c2-6132-4b7a-9ac6-854e65729e03"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:23:37.535974 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.535953    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b545c8c2-6132-4b7a-9ac6-854e65729e03-kube-api-access-swb8f" (OuterVolumeSpecName: "kube-api-access-swb8f") pod "b545c8c2-6132-4b7a-9ac6-854e65729e03" (UID: "b545c8c2-6132-4b7a-9ac6-854e65729e03"). InnerVolumeSpecName "kube-api-access-swb8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:23:37.634583 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.634537    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b545c8c2-6132-4b7a-9ac6-854e65729e03-plugins-config-volume\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:23:37.634583 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.634573    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swb8f\" (UniqueName: \"kubernetes.io/projected/b545c8c2-6132-4b7a-9ac6-854e65729e03-kube-api-access-swb8f\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:23:37.909487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:37.909396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:38.000982 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.000947    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:38.116534 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.116499    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:38.200669 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.200583    2576 generic.go:356] "Generic (PLEG): container finished" podID="b545c8c2-6132-4b7a-9ac6-854e65729e03" containerID="e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83" exitCode=0
Sep 04 22:23:38.200669 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.200624    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"
Sep 04 22:23:38.200669 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.200637    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg" event={"ID":"b545c8c2-6132-4b7a-9ac6-854e65729e03","Type":"ContainerDied","Data":"e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83"}
Sep 04 22:23:38.200928 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.200687    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg" event={"ID":"b545c8c2-6132-4b7a-9ac6-854e65729e03","Type":"ContainerDied","Data":"042ef37d5d53a0745b0c622cf9631f530cf5892039471ac9435d7b5ad66f5b31"}
Sep 04 22:23:38.200928 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.200710    2576 scope.go:117] "RemoveContainer" containerID="e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83"
Sep 04 22:23:38.209961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.209942    2576 scope.go:117] "RemoveContainer" containerID="e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83"
Sep 04 22:23:38.210265 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:23:38.210248    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83\": container with ID starting with e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83 not found: ID does not exist" containerID="e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83"
Sep 04 22:23:38.210331 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.210274    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83"} err="failed to get container status \"e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83\": rpc error: code = NotFound desc = could not find container \"e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83\": container with ID starting with e87b77f0b6e91f4f787b35b16a90081730fff23eded81726f427df69d30b3b83 not found: ID does not exist"
Sep 04 22:23:38.224711 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.224687    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"]
Sep 04 22:23:38.232904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.232880    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-c561f9fe-59d55b44cd-d5lqg"]
Sep 04 22:23:38.385181 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.385134    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"]
Sep 04 22:23:38.385791 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.385772    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b545c8c2-6132-4b7a-9ac6-854e65729e03" containerName="payload-pre-processing"
Sep 04 22:23:38.385791 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.385792    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b545c8c2-6132-4b7a-9ac6-854e65729e03" containerName="payload-pre-processing"
Sep 04 22:23:38.385954 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.385895    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b545c8c2-6132-4b7a-9ac6-854e65729e03" containerName="payload-pre-processing"
Sep 04 22:23:38.391274 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.391243    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.394067 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.394022    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-bb137412-gw-openshift-default-dockercfg-tdwb9\""
Sep 04 22:23:38.402135 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.402115    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"]
Sep 04 22:23:38.415942 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.415911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:38.441754 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.441722    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-envoy\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.441891 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.441774    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-data\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.441891 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.441871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a2d6d-da51-4038-a276-08736dce449e-istio-podinfo\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.441967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.441911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-istio-token\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.442000 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.441973    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnc9t\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-kube-api-access-dnc9t\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.442038 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.442008    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-credential-socket\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.442102 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.442044    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-socket\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.442141 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.442124    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a2d6d-da51-4038-a276-08736dce449e-istiod-ca-cert\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.442176 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.442167    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-certs\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.543423 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.543375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-credential-socket\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.543636 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.543436    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-socket\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.543636 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.543468    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a2d6d-da51-4038-a276-08736dce449e-istiod-ca-cert\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.543636 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.543500    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-certs\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.543636 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.543543    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-envoy\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.543636 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.543582    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-data\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.543636 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.543622    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a2d6d-da51-4038-a276-08736dce449e-istio-podinfo\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.543949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.543805    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-credential-socket\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.543949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.543865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-socket\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.544072 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.544036    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-certs\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.544138 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.544071    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-istio-token\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.544138 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.544129    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-data\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.544236 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.544160    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnc9t\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-kube-api-access-dnc9t\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.544462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.544440    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a2d6d-da51-4038-a276-08736dce449e-istiod-ca-cert\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.546370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.546343    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-envoy\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.546676 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.546646    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a2d6d-da51-4038-a276-08736dce449e-istio-podinfo\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.553482 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.553456    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-istio-token\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.553823 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.553802    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnc9t\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-kube-api-access-dnc9t\") pod \"e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") " pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.704989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.704951    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:38.880548 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.880462    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:38.918137 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:38.918104    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b545c8c2-6132-4b7a-9ac6-854e65729e03" path="/var/lib/kubelet/pods/b545c8c2-6132-4b7a-9ac6-854e65729e03/volumes"
Sep 04 22:23:39.009355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:39.009320    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"]
Sep 04 22:23:39.012476 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:23:39.012444    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78a2d6d_da51_4038_a276_08736dce449e.slice/crio-b1edc079d0ed4e77a12d3b541760fb3eba41056649cb86887cbb6b044099c80d WatchSource:0}: Error finding container b1edc079d0ed4e77a12d3b541760fb3eba41056649cb86887cbb6b044099c80d: Status 404 returned error can't find the container with id b1edc079d0ed4e77a12d3b541760fb3eba41056649cb86887cbb6b044099c80d
Sep 04 22:23:39.014844 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:39.014809    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:39.014936 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:39.014880    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:39.014936 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:39.014909    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:39.209963 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:39.209925    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" event={"ID":"b78a2d6d-da51-4038-a276-08736dce449e","Type":"ContainerStarted","Data":"3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744"}
Sep 04 22:23:39.210159 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:39.209988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" event={"ID":"b78a2d6d-da51-4038-a276-08736dce449e","Type":"ContainerStarted","Data":"b1edc079d0ed4e77a12d3b541760fb3eba41056649cb86887cbb6b044099c80d"}
Sep 04 22:23:39.242372 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:39.242316    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" podStartSLOduration=1.242298149 podStartE2EDuration="1.242298149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:23:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:23:39.239698649 +0000 UTC m=+902.758421178" watchObservedRunningTime="2026-09-04 22:23:39.242298149 +0000 UTC m=+902.761020682"
Sep 04 22:23:39.705322 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:39.705287    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:40.711439 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:40.711407    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:40.715177 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:40.715144    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:40.818163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:40.818121    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:40.919970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:40.919918    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:41.121756 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:41.121656    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:41.221546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:41.221512    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:41.222622 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:41.222601    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:23:41.564900 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:41.564858    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:41.665527 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:41.665496    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:43.439975 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:43.439886    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:44.239168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:44.239136    2576 generic.go:356] "Generic (PLEG): container finished" podID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerID="e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145" exitCode=0
Sep 04 22:23:44.239347 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:44.239209    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" event={"ID":"dd604e6d-a4d9-4787-b133-f925d2b84367","Type":"ContainerDied","Data":"e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145"}
Sep 04 22:23:44.239701 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:44.239682    2576 scope.go:117] "RemoveContainer" containerID="e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145"
Sep 04 22:23:44.240806 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:44.240776    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:44.240857 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:44.240849    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:44.240893 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:44.240879    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:23:45.018158 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:45.018121    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:23:45.246246 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:45.246206    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" event={"ID":"dd604e6d-a4d9-4787-b133-f925d2b84367","Type":"ContainerStarted","Data":"d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7"}
Sep 04 22:23:45.535960 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:45.535917    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:45.537253 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:45.537225    2576 patch_prober.go:28] interesting pod/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 04 22:23:45.537380 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:45.537271    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 04 22:23:46.536336 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:46.536291    2576 patch_prober.go:28] interesting pod/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 04 22:23:46.536827 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:46.536369    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 04 22:23:47.535478 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:47.535441    2576 patch_prober.go:28] interesting pod/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 04 22:23:47.535721 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:47.535508    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 04 22:23:48.525501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:48.525446    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:23:48.535780 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:48.535737    2576 patch_prober.go:28] interesting pod/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 04 22:23:48.535978 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:48.535803    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 04 22:23:49.084673 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:49.084630    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:49.535314 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:49.535281    2576 patch_prober.go:28] interesting pod/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused" start-of-body=
Sep 04 22:23:49.535855 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:49.535346    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.41:15021/healthz/ready\": dial tcp 10.133.0.41:15021: connect: connection refused"
Sep 04 22:23:50.539756 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:50.539725    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:50.540856 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:50.540833    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:23:57.896466 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:23:57.896415    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:00.343146 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:00.343113    2576 generic.go:356] "Generic (PLEG): container finished" podID="b78a2d6d-da51-4038-a276-08736dce449e" containerID="3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744" exitCode=0
Sep 04 22:24:00.343611 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:00.343187    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" event={"ID":"b78a2d6d-da51-4038-a276-08736dce449e","Type":"ContainerDied","Data":"3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744"}
Sep 04 22:24:00.343668 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:00.343654    2576 scope.go:117] "RemoveContainer" containerID="3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744"
Sep 04 22:24:00.344803 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:00.344775    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:24:00.344861 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:00.344853    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:24:00.344902 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:00.344886    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:24:01.350678 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:01.350636    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" event={"ID":"b78a2d6d-da51-4038-a276-08736dce449e","Type":"ContainerStarted","Data":"cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813"}
Sep 04 22:24:01.705616 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:01.705567    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:24:01.705820 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:01.705802    2576 patch_prober.go:28] interesting pod/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 04 22:24:01.705864 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:01.705843    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 04 22:24:01.933739 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:01.933701    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:02.154557 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:02.154464    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:02.705808 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:02.705772    2576 patch_prober.go:28] interesting pod/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 04 22:24:02.706242 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:02.705842    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 04 22:24:03.706218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:03.706179    2576 patch_prober.go:28] interesting pod/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 04 22:24:03.706665 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:03.706238    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 04 22:24:04.705802 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:04.705765    2576 patch_prober.go:28] interesting pod/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 04 22:24:04.705995 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:04.705831    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 04 22:24:05.445495 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:05.445457    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:24:05.706008 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:05.705976    2576 patch_prober.go:28] interesting pod/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused" start-of-body=
Sep 04 22:24:05.706196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:05.706100    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.43:15021/healthz/ready\": dial tcp 10.133.0.43:15021: connect: connection refused"
Sep 04 22:24:06.875996 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:06.875956    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:24:06.876494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:06.876381    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:24:06.877358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:06.877337    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:24:12.399488 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:12.399451    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:12.501534 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:12.501464    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:12.589264 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:12.589226    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:12.650393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:12.650292    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:12.743613 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:12.743571    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"]
Sep 04 22:24:12.743863 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:12.743834    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy" containerID="cri-o://cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813" gracePeriod=30
Sep 04 22:24:12.769431 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:12.769395    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:13.199574 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:13.199538    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:13.532547 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:13.532508    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:13.692696 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:13.692649    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:16.392187 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:16.392138    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:16.484987 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:16.484956    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:17.565520 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:17.565470    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:17.666642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:17.666604    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:18.007996 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.007972    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:24:18.163380 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163293    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-credential-socket\") pod \"b78a2d6d-da51-4038-a276-08736dce449e\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") "
Sep 04 22:24:18.163380 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163334    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-certs\") pod \"b78a2d6d-da51-4038-a276-08736dce449e\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") "
Sep 04 22:24:18.163380 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163365    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-data\") pod \"b78a2d6d-da51-4038-a276-08736dce449e\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") "
Sep 04 22:24:18.163653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163465    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dnc9t\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-kube-api-access-dnc9t\") pod \"b78a2d6d-da51-4038-a276-08736dce449e\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") "
Sep 04 22:24:18.163653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163504    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a2d6d-da51-4038-a276-08736dce449e-istiod-ca-cert\") pod \"b78a2d6d-da51-4038-a276-08736dce449e\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") "
Sep 04 22:24:18.163653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163537    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-istio-token\") pod \"b78a2d6d-da51-4038-a276-08736dce449e\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") "
Sep 04 22:24:18.163653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163565    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-envoy\") pod \"b78a2d6d-da51-4038-a276-08736dce449e\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") "
Sep 04 22:24:18.163653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163609    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a2d6d-da51-4038-a276-08736dce449e-istio-podinfo\") pod \"b78a2d6d-da51-4038-a276-08736dce449e\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") "
Sep 04 22:24:18.163653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163630    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b78a2d6d-da51-4038-a276-08736dce449e" (UID: "b78a2d6d-da51-4038-a276-08736dce449e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:24:18.163952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163675    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-socket\") pod \"b78a2d6d-da51-4038-a276-08736dce449e\" (UID: \"b78a2d6d-da51-4038-a276-08736dce449e\") "
Sep 04 22:24:18.163952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163707    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-data" (OuterVolumeSpecName: "istio-data") pod "b78a2d6d-da51-4038-a276-08736dce449e" (UID: "b78a2d6d-da51-4038-a276-08736dce449e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:24:18.164083 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.163955    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b78a2d6d-da51-4038-a276-08736dce449e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b78a2d6d-da51-4038-a276-08736dce449e" (UID: "b78a2d6d-da51-4038-a276-08736dce449e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:24:18.164302 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.164259    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b78a2d6d-da51-4038-a276-08736dce449e" (UID: "b78a2d6d-da51-4038-a276-08736dce449e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:24:18.164302 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.164280    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78a2d6d-da51-4038-a276-08736dce449e-istiod-ca-cert\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:18.164302 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.164301    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:18.164517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.164314    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-data\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:18.164517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.164371    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b78a2d6d-da51-4038-a276-08736dce449e" (UID: "b78a2d6d-da51-4038-a276-08736dce449e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:24:18.166490 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.166431    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b78a2d6d-da51-4038-a276-08736dce449e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b78a2d6d-da51-4038-a276-08736dce449e" (UID: "b78a2d6d-da51-4038-a276-08736dce449e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 22:24:18.166490 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.166440    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-istio-token" (OuterVolumeSpecName: "istio-token") pod "b78a2d6d-da51-4038-a276-08736dce449e" (UID: "b78a2d6d-da51-4038-a276-08736dce449e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:24:18.166490 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.166439    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-kube-api-access-dnc9t" (OuterVolumeSpecName: "kube-api-access-dnc9t") pod "b78a2d6d-da51-4038-a276-08736dce449e" (UID: "b78a2d6d-da51-4038-a276-08736dce449e"). InnerVolumeSpecName "kube-api-access-dnc9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:24:18.166676 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.166531    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b78a2d6d-da51-4038-a276-08736dce449e" (UID: "b78a2d6d-da51-4038-a276-08736dce449e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:24:18.194426 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.194383    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:18.264990 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.264950    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-istio-token\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:18.264990 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.264979    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-istio-envoy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:18.264990 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.264988    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78a2d6d-da51-4038-a276-08736dce449e-istio-podinfo\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:18.264990 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.264997    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-workload-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:18.265275 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.265005    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78a2d6d-da51-4038-a276-08736dce449e-credential-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:18.265275 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.265015    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnc9t\" (UniqueName: \"kubernetes.io/projected/b78a2d6d-da51-4038-a276-08736dce449e-kube-api-access-dnc9t\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:18.287644 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.287607    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:18.324013 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.323972    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"]
Sep 04 22:24:18.324486 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.324467    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy"
Sep 04 22:24:18.324587 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.324488    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy"
Sep 04 22:24:18.324641 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.324607    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy"
Sep 04 22:24:18.328169 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.328144    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:18.333335 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.333312    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-20c0448a\""
Sep 04 22:24:18.333830 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.333813    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-20c0448a-dockercfg-qt692\""
Sep 04 22:24:18.340206 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.340184    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"]
Sep 04 22:24:18.453483 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.453457    2576 generic.go:356] "Generic (PLEG): container finished" podID="b78a2d6d-da51-4038-a276-08736dce449e" containerID="cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813" exitCode=0
Sep 04 22:24:18.453638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.453505    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"
Sep 04 22:24:18.453638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.453532    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" event={"ID":"b78a2d6d-da51-4038-a276-08736dce449e","Type":"ContainerDied","Data":"cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813"}
Sep 04 22:24:18.453638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.453577    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b" event={"ID":"b78a2d6d-da51-4038-a276-08736dce449e","Type":"ContainerDied","Data":"b1edc079d0ed4e77a12d3b541760fb3eba41056649cb86887cbb6b044099c80d"}
Sep 04 22:24:18.453638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.453602    2576 scope.go:117] "RemoveContainer" containerID="cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813"
Sep 04 22:24:18.463627 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.463607    2576 scope.go:117] "RemoveContainer" containerID="3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744"
Sep 04 22:24:18.467215 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.467191    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e5c873-425e-4e0b-a399-e117b45d4e36-plugins-config-volume\") pod \"payload-processing-e2e-ait-20c0448a-dffbc-skrhf\" (UID: \"67e5c873-425e-4e0b-a399-e117b45d4e36\") " pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:18.467333 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.467232    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnlgs\" (UniqueName: \"kubernetes.io/projected/67e5c873-425e-4e0b-a399-e117b45d4e36-kube-api-access-hnlgs\") pod \"payload-processing-e2e-ait-20c0448a-dffbc-skrhf\" (UID: \"67e5c873-425e-4e0b-a399-e117b45d4e36\") " pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:18.472184 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.472165    2576 scope.go:117] "RemoveContainer" containerID="cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813"
Sep 04 22:24:18.472444 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:24:18.472426    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813\": container with ID starting with cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813 not found: ID does not exist" containerID="cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813"
Sep 04 22:24:18.472501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.472451    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813"} err="failed to get container status \"cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813\": rpc error: code = NotFound desc = could not find container \"cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813\": container with ID starting with cf2f9b3432c7a34980873ed3ab1017dd424dc9cf25b6f3b077078a8c5b419813 not found: ID does not exist"
Sep 04 22:24:18.472501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.472469    2576 scope.go:117] "RemoveContainer" containerID="3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744"
Sep 04 22:24:18.472696 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:24:18.472679    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744\": container with ID starting with 3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744 not found: ID does not exist" containerID="3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744"
Sep 04 22:24:18.472739 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.472701    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744"} err="failed to get container status \"3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744\": rpc error: code = NotFound desc = could not find container \"3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744\": container with ID starting with 3909438d70a506ac70e29e0fb78ca529922af10c5ce218d9411d96c1bb234744 not found: ID does not exist"
Sep 04 22:24:18.479202 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.479178    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"]
Sep 04 22:24:18.482168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.482147    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-bb137412-gw-openshift-default-85dc8df956-qcv2b"]
Sep 04 22:24:18.568301 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.568269    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e5c873-425e-4e0b-a399-e117b45d4e36-plugins-config-volume\") pod \"payload-processing-e2e-ait-20c0448a-dffbc-skrhf\" (UID: \"67e5c873-425e-4e0b-a399-e117b45d4e36\") " pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:18.568714 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.568310    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnlgs\" (UniqueName: \"kubernetes.io/projected/67e5c873-425e-4e0b-a399-e117b45d4e36-kube-api-access-hnlgs\") pod \"payload-processing-e2e-ait-20c0448a-dffbc-skrhf\" (UID: \"67e5c873-425e-4e0b-a399-e117b45d4e36\") " pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:18.568940 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.568919    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e5c873-425e-4e0b-a399-e117b45d4e36-plugins-config-volume\") pod \"payload-processing-e2e-ait-20c0448a-dffbc-skrhf\" (UID: \"67e5c873-425e-4e0b-a399-e117b45d4e36\") " pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:18.577091 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.577043    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnlgs\" (UniqueName: \"kubernetes.io/projected/67e5c873-425e-4e0b-a399-e117b45d4e36-kube-api-access-hnlgs\") pod \"payload-processing-e2e-ait-20c0448a-dffbc-skrhf\" (UID: \"67e5c873-425e-4e0b-a399-e117b45d4e36\") " pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:18.612471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.612445    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:18.641282 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.641249    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:18.915813 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:24:18.915780    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e5c873_425e_4e0b_a399_e117b45d4e36.slice/crio-b4b574d1bd8c6ffba0a4425882fc8f091f141b1c2f39b8d234c327ba77d8d05d WatchSource:0}: Error finding container b4b574d1bd8c6ffba0a4425882fc8f091f141b1c2f39b8d234c327ba77d8d05d: Status 404 returned error can't find the container with id b4b574d1bd8c6ffba0a4425882fc8f091f141b1c2f39b8d234c327ba77d8d05d
Sep 04 22:24:18.919172 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.919147    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b78a2d6d-da51-4038-a276-08736dce449e" path="/var/lib/kubelet/pods/b78a2d6d-da51-4038-a276-08736dce449e/volumes"
Sep 04 22:24:18.919488 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:18.919470    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"]
Sep 04 22:24:19.461797 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:19.461764    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf" event={"ID":"67e5c873-425e-4e0b-a399-e117b45d4e36","Type":"ContainerStarted","Data":"19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa"}
Sep 04 22:24:19.462000 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:19.461811    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf" event={"ID":"67e5c873-425e-4e0b-a399-e117b45d4e36","Type":"ContainerStarted","Data":"b4b574d1bd8c6ffba0a4425882fc8f091f141b1c2f39b8d234c327ba77d8d05d"}
Sep 04 22:24:19.462000 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:19.461888    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:19.482282 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:19.482231    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf" podStartSLOduration=1.4822159529999999 podStartE2EDuration="1.482215953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:24:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:24:19.480450005 +0000 UTC m=+942.999172541" watchObservedRunningTime="2026-09-04 22:24:19.482215953 +0000 UTC m=+943.000938486"
Sep 04 22:24:23.101875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:23.101829    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:24:30.468788 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:30.468753    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:31.580062 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.580018    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"]
Sep 04 22:24:31.580480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.580467    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy"
Sep 04 22:24:31.580526 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.580481    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy"
Sep 04 22:24:31.580571 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.580561    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b78a2d6d-da51-4038-a276-08736dce449e" containerName="istio-proxy"
Sep 04 22:24:31.584050 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.584032    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.587943 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.587914    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a8d845\"/\"default-dockercfg-9kdcs\""
Sep 04 22:24:31.587943 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.587930    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a8d845\"/\"openshift-service-ca.crt\""
Sep 04 22:24:31.588192 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.587921    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-a8d845\"/\"autores-model-a8d845-kserve-self-signed-certs\""
Sep 04 22:24:31.588192 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.587922    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-a8d845\"/\"kube-root-ca.crt\""
Sep 04 22:24:31.594099 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.594077    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"]
Sep 04 22:24:31.597767 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.597743    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69881682-7640-4077-8ff4-836f62457440-tls-certs\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.597892 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.597822    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlq5h\" (UniqueName: \"kubernetes.io/projected/69881682-7640-4077-8ff4-836f62457440-kube-api-access-mlq5h\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.597892 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.597855    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-model-cache\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.597892 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.597871    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-tmp-dir\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.598047 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.597949    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-dshm\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.598047 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.597975    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-home\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.698680 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.698634    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-dshm\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.698845 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.698684    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-home\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.698845 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.698739    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69881682-7640-4077-8ff4-836f62457440-tls-certs\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.698845 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.698825    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlq5h\" (UniqueName: \"kubernetes.io/projected/69881682-7640-4077-8ff4-836f62457440-kube-api-access-mlq5h\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.699011 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.698975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-model-cache\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.699193 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.699022    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-tmp-dir\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.699193 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.699183    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-home\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.699339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.699319    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-model-cache\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.699404 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.699388    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-tmp-dir\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.700877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.700857    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-dshm\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.701283 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.701261    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69881682-7640-4077-8ff4-836f62457440-tls-certs\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.708642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.708620    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlq5h\" (UniqueName: \"kubernetes.io/projected/69881682-7640-4077-8ff4-836f62457440-kube-api-access-mlq5h\") pod \"autores-model-a8d845-kserve-566c9598b7-4fpsx\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") " pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:31.893519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.893424    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:31.893661 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:31.893519    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:32.166492 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:32.166414    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"]
Sep 04 22:24:32.169766 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:24:32.169740    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod69881682_7640_4077_8ff4_836f62457440.slice/crio-b6b9e1d67461a31f65c41dc11b688bd6f85ca5d82439f671a50e000ad3fa1a6f WatchSource:0}: Error finding container b6b9e1d67461a31f65c41dc11b688bd6f85ca5d82439f671a50e000ad3fa1a6f: Status 404 returned error can't find the container with id b6b9e1d67461a31f65c41dc11b688bd6f85ca5d82439f671a50e000ad3fa1a6f
Sep 04 22:24:32.534953 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:32.534903    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx" event={"ID":"69881682-7640-4077-8ff4-836f62457440","Type":"ContainerStarted","Data":"783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7"}
Sep 04 22:24:32.534953 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:32.534965    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx" event={"ID":"69881682-7640-4077-8ff4-836f62457440","Type":"ContainerStarted","Data":"b6b9e1d67461a31f65c41dc11b688bd6f85ca5d82439f671a50e000ad3fa1a6f"}
Sep 04 22:24:32.553412 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:32.553341    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx" podStartSLOduration=1.553326135 podStartE2EDuration="1.553326135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:24:32.552493332 +0000 UTC m=+956.071215863" watchObservedRunningTime="2026-09-04 22:24:32.553326135 +0000 UTC m=+956.072048678"
Sep 04 22:24:41.894541 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:41.894495    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:41.894541 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:41.894547    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:41.906657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:41.906633    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:42.596344 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:42.596315    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:24:43.297272 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:43.297237    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:24:48.166469 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.166432    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:48.190655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.190624    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"]
Sep 04 22:24:48.190912 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.190871    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf" podUID="67e5c873-425e-4e0b-a399-e117b45d4e36" containerName="payload-processing" containerID="cri-o://19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa" gracePeriod=30
Sep 04 22:24:48.264487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.264454    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:48.369499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.369422    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:48.438559 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.438533    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:48.458459 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.458425    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:48.569092 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.569038    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e5c873-425e-4e0b-a399-e117b45d4e36-plugins-config-volume\") pod \"67e5c873-425e-4e0b-a399-e117b45d4e36\" (UID: \"67e5c873-425e-4e0b-a399-e117b45d4e36\") "
Sep 04 22:24:48.569287 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.569129    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hnlgs\" (UniqueName: \"kubernetes.io/projected/67e5c873-425e-4e0b-a399-e117b45d4e36-kube-api-access-hnlgs\") pod \"67e5c873-425e-4e0b-a399-e117b45d4e36\" (UID: \"67e5c873-425e-4e0b-a399-e117b45d4e36\") "
Sep 04 22:24:48.569504 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.569471    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e5c873-425e-4e0b-a399-e117b45d4e36-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "67e5c873-425e-4e0b-a399-e117b45d4e36" (UID: "67e5c873-425e-4e0b-a399-e117b45d4e36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:24:48.571183 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.571158    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e5c873-425e-4e0b-a399-e117b45d4e36-kube-api-access-hnlgs" (OuterVolumeSpecName: "kube-api-access-hnlgs") pod "67e5c873-425e-4e0b-a399-e117b45d4e36" (UID: "67e5c873-425e-4e0b-a399-e117b45d4e36"). InnerVolumeSpecName "kube-api-access-hnlgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:24:48.625435 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.625409    2576 generic.go:356] "Generic (PLEG): container finished" podID="67e5c873-425e-4e0b-a399-e117b45d4e36" containerID="19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa" exitCode=0
Sep 04 22:24:48.625535 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.625448    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"
Sep 04 22:24:48.625578 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.625448    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf" event={"ID":"67e5c873-425e-4e0b-a399-e117b45d4e36","Type":"ContainerDied","Data":"19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa"}
Sep 04 22:24:48.625623 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.625579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf" event={"ID":"67e5c873-425e-4e0b-a399-e117b45d4e36","Type":"ContainerDied","Data":"b4b574d1bd8c6ffba0a4425882fc8f091f141b1c2f39b8d234c327ba77d8d05d"}
Sep 04 22:24:48.625623 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.625607    2576 scope.go:117] "RemoveContainer" containerID="19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa"
Sep 04 22:24:48.634507 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.634491    2576 scope.go:117] "RemoveContainer" containerID="19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa"
Sep 04 22:24:48.634770 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:24:48.634751    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa\": container with ID starting with 19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa not found: ID does not exist" containerID="19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa"
Sep 04 22:24:48.634836 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.634778    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa"} err="failed to get container status \"19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa\": rpc error: code = NotFound desc = could not find container \"19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa\": container with ID starting with 19e07f3bd0485032dd302d1a4c3737a5eea6daa9f52bdc029d5b2662b97d54fa not found: ID does not exist"
Sep 04 22:24:48.652191 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.652167    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"]
Sep 04 22:24:48.655767 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.655749    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-20c0448a-dffbc-skrhf"]
Sep 04 22:24:48.670702 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.670648    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67e5c873-425e-4e0b-a399-e117b45d4e36-plugins-config-volume\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:48.670702 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.670696    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnlgs\" (UniqueName: \"kubernetes.io/projected/67e5c873-425e-4e0b-a399-e117b45d4e36-kube-api-access-hnlgs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:24:48.917986 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:48.917944    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67e5c873-425e-4e0b-a399-e117b45d4e36" path="/var/lib/kubelet/pods/67e5c873-425e-4e0b-a399-e117b45d4e36/volumes"
Sep 04 22:24:49.488166 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:49.488128    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:49.912718 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:49.912631    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:50.218394 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:50.218350    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:50.518800 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:50.518705    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:52.615110 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:52.615047    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:52.708237 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:52.708202    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:53.899221 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:53.899177    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"]
Sep 04 22:24:53.899700 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:53.899668    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67e5c873-425e-4e0b-a399-e117b45d4e36" containerName="payload-processing"
Sep 04 22:24:53.899700 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:53.899682    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="67e5c873-425e-4e0b-a399-e117b45d4e36" containerName="payload-processing"
Sep 04 22:24:53.899838 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:53.899773    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="67e5c873-425e-4e0b-a399-e117b45d4e36" containerName="payload-processing"
Sep 04 22:24:53.904318 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:53.904298    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:53.908245 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:53.908222    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-25fea002-serving-cert\""
Sep 04 22:24:53.915671 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:53.915651    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"]
Sep 04 22:24:53.940319 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:53.940267    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:54.010501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.010469    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:54.032715 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.032671    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-openshift-service-ca\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.032919 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.032752    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-maas-api-tls\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.032919 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.032813    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c74rz\" (UniqueName: \"kubernetes.io/projected/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-kube-api-access-c74rz\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.033030 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.032911    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-metrics-certs\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.134098 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.134061    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-maas-api-tls\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.134299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.134137    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c74rz\" (UniqueName: \"kubernetes.io/projected/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-kube-api-access-c74rz\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.134299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.134169    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-metrics-certs\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.134299 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:24:54.134196    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-serving-cert: secret "maas-api-e2e-derive-25fea002-serving-cert" not found
Sep 04 22:24:54.134299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.134241    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-openshift-service-ca\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.134299 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:24:54.134274    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-maas-api-tls podName:549cdb9f-d3b8-42d2-9050-8ac1130ebcf5 nodeName:}" failed. No retries permitted until 2026-09-04 22:24:54.634256114 +0000 UTC m=+978.152978625 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-maas-api-tls") pod "maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45" (UID: "549cdb9f-d3b8-42d2-9050-8ac1130ebcf5") : secret "maas-api-e2e-derive-25fea002-serving-cert" not found
Sep 04 22:24:54.134887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.134869    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-openshift-service-ca\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.136817 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.136792    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-metrics-certs\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.142572 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.142546    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c74rz\" (UniqueName: \"kubernetes.io/projected/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-kube-api-access-c74rz\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.300989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.300943    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:54.640244 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.640147    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-maas-api-tls\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.642493 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.642469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-maas-api-tls\") pod \"maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:54.819064 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:54.819033    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:55.093379 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:55.093343    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"]
Sep 04 22:24:55.096094 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:24:55.096041    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod549cdb9f_d3b8_42d2_9050_8ac1130ebcf5.slice/crio-1f98b105db2772e566022fbd5cf88ee113949bcfad72c3e2ef4e03e770066a64 WatchSource:0}: Error finding container 1f98b105db2772e566022fbd5cf88ee113949bcfad72c3e2ef4e03e770066a64: Status 404 returned error can't find the container with id 1f98b105db2772e566022fbd5cf88ee113949bcfad72c3e2ef4e03e770066a64
Sep 04 22:24:55.398536 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:55.398458    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:24:55.663279 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:55.663184    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45" event={"ID":"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5","Type":"ContainerStarted","Data":"b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb"}
Sep 04 22:24:55.663279 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:55.663238    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45" event={"ID":"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5","Type":"ContainerStarted","Data":"1f98b105db2772e566022fbd5cf88ee113949bcfad72c3e2ef4e03e770066a64"}
Sep 04 22:24:55.663477 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:55.663313    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:24:55.680472 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:24:55.680417    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45" podStartSLOduration=2.369671049 podStartE2EDuration="2.680400345s" totalImagesPullingTime="310.729296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:24:55.679910153 +0000 UTC m=+979.198632686" watchObservedRunningTime="2026-09-04 22:24:55.680400345 +0000 UTC m=+979.199122877"
Sep 04 22:25:01.674588 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:01.674504    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:25:02.996810 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:02.996772    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:25:04.538281 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:04.538245    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:07.628302 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:07.628268    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:15.037587 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:15.037553    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:17.675022 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:17.674983    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:18.519160 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:18.519113    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:19.296004 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.295964    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"]
Sep 04 22:25:19.296474 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.296203    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw" podUID="7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" containerName="maas-api" containerID="cri-o://0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48" gracePeriod=30
Sep 04 22:25:19.332968 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.332928    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:19.430573 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.430544    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:19.532062 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.532015    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:19.549166 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.549113    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:25:19.630858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.630829    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:19.679764 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.679726    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-metrics-certs\") pod \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") "
Sep 04 22:25:19.679922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.679794    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9j56w\" (UniqueName: \"kubernetes.io/projected/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-kube-api-access-9j56w\") pod \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") "
Sep 04 22:25:19.679922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.679861    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-maas-api-tls\") pod \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") "
Sep 04 22:25:19.679922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.679903    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-openshift-service-ca\") pod \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\" (UID: \"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a\") "
Sep 04 22:25:19.680515 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.680485    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" (UID: "7bf143ca-6576-4eaa-bb15-425cfb0b6a9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:25:19.682373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.682347    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" (UID: "7bf143ca-6576-4eaa-bb15-425cfb0b6a9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:25:19.682546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.682522    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" (UID: "7bf143ca-6576-4eaa-bb15-425cfb0b6a9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:25:19.682598 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.682549    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-kube-api-access-9j56w" (OuterVolumeSpecName: "kube-api-access-9j56w") pod "7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" (UID: "7bf143ca-6576-4eaa-bb15-425cfb0b6a9a"). InnerVolumeSpecName "kube-api-access-9j56w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:25:19.781266 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.781230    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:19.781266 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.781259    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:19.781471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.781276    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9j56w\" (UniqueName: \"kubernetes.io/projected/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-kube-api-access-9j56w\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:19.781471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.781288    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:19.792841 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.792807    2576 generic.go:356] "Generic (PLEG): container finished" podID="7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" containerID="0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48" exitCode=0
Sep 04 22:25:19.792988 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.792874    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"
Sep 04 22:25:19.792988 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.792885    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw" event={"ID":"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a","Type":"ContainerDied","Data":"0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48"}
Sep 04 22:25:19.792988 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.792947    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw" event={"ID":"7bf143ca-6576-4eaa-bb15-425cfb0b6a9a","Type":"ContainerDied","Data":"348bcca4e4ab550a932d9cb42cbbaefa5bf22f8c5e7a04df6e2bdd261cec0b6c"}
Sep 04 22:25:19.792988 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.792973    2576 scope.go:117] "RemoveContainer" containerID="0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48"
Sep 04 22:25:19.801779 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.801761    2576 scope.go:117] "RemoveContainer" containerID="0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48"
Sep 04 22:25:19.802048 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:25:19.802031    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48\": container with ID starting with 0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48 not found: ID does not exist" containerID="0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48"
Sep 04 22:25:19.802155 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.802068    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48"} err="failed to get container status \"0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48\": rpc error: code = NotFound desc = could not find container \"0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48\": container with ID starting with 0cf94d9ae3cefc691eec98437f46fe83468d8b5507d3da1b918794918befaf48 not found: ID does not exist"
Sep 04 22:25:19.818233 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.818210    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"]
Sep 04 22:25:19.823219 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.823196    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-a8d845-cf74469cd-94wbw"]
Sep 04 22:25:19.919262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:19.919230    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:20.924430 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:20.924401    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" path="/var/lib/kubelet/pods/7bf143ca-6576-4eaa-bb15-425cfb0b6a9a/volumes"
Sep 04 22:25:22.638771 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:22.638733    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:24.327550 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.327516    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"]
Sep 04 22:25:24.327997 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.327757    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45" podUID="549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" containerName="maas-api" containerID="cri-o://b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb" gracePeriod=30
Sep 04 22:25:24.357785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.357755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:24.458964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.458935    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:24.559275 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.559244    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:24.574755 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.574734    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:25:24.630766 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.630692    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c74rz\" (UniqueName: \"kubernetes.io/projected/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-kube-api-access-c74rz\") pod \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") "
Sep 04 22:25:24.630766 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.630747    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-metrics-certs\") pod \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") "
Sep 04 22:25:24.630982 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.630796    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-maas-api-tls\") pod \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") "
Sep 04 22:25:24.630982 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.630839    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-openshift-service-ca\") pod \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\" (UID: \"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5\") "
Sep 04 22:25:24.631265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.631238    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" (UID: "549cdb9f-d3b8-42d2-9050-8ac1130ebcf5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:25:24.632941 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.632912    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" (UID: "549cdb9f-d3b8-42d2-9050-8ac1130ebcf5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:25:24.633040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.632943    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" (UID: "549cdb9f-d3b8-42d2-9050-8ac1130ebcf5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:25:24.633040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.632951    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-kube-api-access-c74rz" (OuterVolumeSpecName: "kube-api-access-c74rz") pod "549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" (UID: "549cdb9f-d3b8-42d2-9050-8ac1130ebcf5"). InnerVolumeSpecName "kube-api-access-c74rz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:25:24.655758 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.655729    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:24.731568 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.731537    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:24.731568 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.731566    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:24.731757 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.731582    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c74rz\" (UniqueName: \"kubernetes.io/projected/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-kube-api-access-c74rz\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:24.731757 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.731595    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:24.744342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.744299    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:24.822968 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.822933    2576 generic.go:356] "Generic (PLEG): container finished" podID="549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" containerID="b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb" exitCode=0
Sep 04 22:25:24.823183 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.822983    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"
Sep 04 22:25:24.823183 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.823011    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45" event={"ID":"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5","Type":"ContainerDied","Data":"b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb"}
Sep 04 22:25:24.823183 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.823079    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45" event={"ID":"549cdb9f-d3b8-42d2-9050-8ac1130ebcf5","Type":"ContainerDied","Data":"1f98b105db2772e566022fbd5cf88ee113949bcfad72c3e2ef4e03e770066a64"}
Sep 04 22:25:24.823183 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.823104    2576 scope.go:117] "RemoveContainer" containerID="b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb"
Sep 04 22:25:24.832729 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.832710    2576 scope.go:117] "RemoveContainer" containerID="b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb"
Sep 04 22:25:24.833007 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:25:24.832988    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb\": container with ID starting with b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb not found: ID does not exist" containerID="b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb"
Sep 04 22:25:24.833072 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.833014    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb"} err="failed to get container status \"b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb\": rpc error: code = NotFound desc = could not find container \"b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb\": container with ID starting with b2cedd6280841fc062e10c96c7f3c1e79d1e219297496c21df3cb253328495bb not found: ID does not exist"
Sep 04 22:25:24.845596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.845573    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"]
Sep 04 22:25:24.853107 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.853081    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-25fea002-6c787d8c6f-bsw45"]
Sep 04 22:25:24.905392 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.905338    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"]
Sep 04 22:25:24.918983 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.918962    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" path="/var/lib/kubelet/pods/549cdb9f-d3b8-42d2-9050-8ac1130ebcf5/volumes"
Sep 04 22:25:24.954512 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:24.954488    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:25.356835 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:25.356801    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:25.457370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:25.457335    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:25.521655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:25.521624    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:25.600007 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:25.599967    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:25:26.857381 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:26.857340    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:32.897044 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.897010    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"]
Sep 04 22:25:32.897441 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.897427    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" containerName="maas-api"
Sep 04 22:25:32.897498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.897442    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" containerName="maas-api"
Sep 04 22:25:32.897498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.897464    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" containerName="maas-api"
Sep 04 22:25:32.897498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.897469    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" containerName="maas-api"
Sep 04 22:25:32.897598 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.897540    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="549cdb9f-d3b8-42d2-9050-8ac1130ebcf5" containerName="maas-api"
Sep 04 22:25:32.897598 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.897549    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bf143ca-6576-4eaa-bb15-425cfb0b6a9a" containerName="maas-api"
Sep 04 22:25:32.902114 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.902092    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:32.905858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.905829    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d9f7ae\"/\"openshift-service-ca.crt\""
Sep 04 22:25:32.905858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.905835    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d9f7ae\"/\"default-dockercfg-gf2xs\""
Sep 04 22:25:32.906024 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.905891    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-d9f7ae\"/\"test-model-d9f7ae-kserve-self-signed-certs\""
Sep 04 22:25:32.906024 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.905937    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-d9f7ae\"/\"kube-root-ca.crt\""
Sep 04 22:25:32.910919 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.910893    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"]
Sep 04 22:25:32.930287 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:32.930255    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:33.017341 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.017299    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26kbr\" (UniqueName: \"kubernetes.io/projected/5f6b9776-f870-421e-afc7-50a71cd347f4-kube-api-access-26kbr\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.017548 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.017511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-model-cache\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.017652 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.017634    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-dshm\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.017725 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.017674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6b9776-f870-421e-afc7-50a71cd347f4-tls-certs\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.017839 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.017821    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-home\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.017963 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.017944    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-tmp-dir\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.118970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.118924    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-model-cache\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.118970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.118977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-dshm\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.119248 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.119008    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6b9776-f870-421e-afc7-50a71cd347f4-tls-certs\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.119248 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.119104    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-home\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.119248 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.119165    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-tmp-dir\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.119248 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.119192    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26kbr\" (UniqueName: \"kubernetes.io/projected/5f6b9776-f870-421e-afc7-50a71cd347f4-kube-api-access-26kbr\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.119502 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.119469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-model-cache\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.119641 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.119524    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-home\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.119641 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.119563    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-tmp-dir\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.121332 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.121306    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-dshm\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.121552 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.121535    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6b9776-f870-421e-afc7-50a71cd347f4-tls-certs\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.133595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.133566    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-26kbr\" (UniqueName: \"kubernetes.io/projected/5f6b9776-f870-421e-afc7-50a71cd347f4-kube-api-access-26kbr\") pod \"test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") " pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.211972 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.211942    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:33.487318 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.487233    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"]
Sep 04 22:25:33.490080 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:25:33.490035    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f6b9776_f870_421e_afc7_50a71cd347f4.slice/crio-775dff686c80c6e071aa12a2e6e8b5698c69d39d4b4c2757ee0b0c014ecc08e2 WatchSource:0}: Error finding container 775dff686c80c6e071aa12a2e6e8b5698c69d39d4b4c2757ee0b0c014ecc08e2: Status 404 returned error can't find the container with id 775dff686c80c6e071aa12a2e6e8b5698c69d39d4b4c2757ee0b0c014ecc08e2
Sep 04 22:25:33.601357 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.601319    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:33.704040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.704000    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:33.803235 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.803149    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:33.877757 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.877723    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x" event={"ID":"5f6b9776-f870-421e-afc7-50a71cd347f4","Type":"ContainerStarted","Data":"c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365"}
Sep 04 22:25:33.877757 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.877761    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x" event={"ID":"5f6b9776-f870-421e-afc7-50a71cd347f4","Type":"ContainerStarted","Data":"775dff686c80c6e071aa12a2e6e8b5698c69d39d4b4c2757ee0b0c014ecc08e2"}
Sep 04 22:25:33.904470 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.904418    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x" podStartSLOduration=1.904402763 podStartE2EDuration="1.904402763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:25:33.903635845 +0000 UTC m=+1017.422358391" watchObservedRunningTime="2026-09-04 22:25:33.904402763 +0000 UTC m=+1017.423125296"
Sep 04 22:25:33.906679 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:33.906643    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:34.315389 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:34.315348    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:36.013595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:36.013554    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:25:39.925476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:39.925420    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx" podUID="69881682-7640-4077-8ff4-836f62457440" containerName="main" containerID="cri-o://783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7" gracePeriod=45
Sep 04 22:25:40.173937 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.173912    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:25:40.194370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194293    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69881682-7640-4077-8ff4-836f62457440-tls-certs\") pod \"69881682-7640-4077-8ff4-836f62457440\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") "
Sep 04 22:25:40.194370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194327    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-dshm\") pod \"69881682-7640-4077-8ff4-836f62457440\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") "
Sep 04 22:25:40.194562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194385    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-model-cache\") pod \"69881682-7640-4077-8ff4-836f62457440\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") "
Sep 04 22:25:40.194562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194443    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-tmp-dir\") pod \"69881682-7640-4077-8ff4-836f62457440\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") "
Sep 04 22:25:40.194562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194467    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-home\") pod \"69881682-7640-4077-8ff4-836f62457440\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") "
Sep 04 22:25:40.194562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194515    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlq5h\" (UniqueName: \"kubernetes.io/projected/69881682-7640-4077-8ff4-836f62457440-kube-api-access-mlq5h\") pod \"69881682-7640-4077-8ff4-836f62457440\" (UID: \"69881682-7640-4077-8ff4-836f62457440\") "
Sep 04 22:25:40.194778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194755    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-model-cache" (OuterVolumeSpecName: "model-cache") pod "69881682-7640-4077-8ff4-836f62457440" (UID: "69881682-7640-4077-8ff4-836f62457440"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:25:40.194826 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194749    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "69881682-7640-4077-8ff4-836f62457440" (UID: "69881682-7640-4077-8ff4-836f62457440"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:25:40.194826 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194783    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-home" (OuterVolumeSpecName: "home") pod "69881682-7640-4077-8ff4-836f62457440" (UID: "69881682-7640-4077-8ff4-836f62457440"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:25:40.194945 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194927    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:40.195001 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194975    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:40.195001 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.194989    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:40.196964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.196928    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-dshm" (OuterVolumeSpecName: "dshm") pod "69881682-7640-4077-8ff4-836f62457440" (UID: "69881682-7640-4077-8ff4-836f62457440"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:25:40.197234 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.197187    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69881682-7640-4077-8ff4-836f62457440-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "69881682-7640-4077-8ff4-836f62457440" (UID: "69881682-7640-4077-8ff4-836f62457440"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:25:40.197234 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.197197    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69881682-7640-4077-8ff4-836f62457440-kube-api-access-mlq5h" (OuterVolumeSpecName: "kube-api-access-mlq5h") pod "69881682-7640-4077-8ff4-836f62457440" (UID: "69881682-7640-4077-8ff4-836f62457440"). InnerVolumeSpecName "kube-api-access-mlq5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:25:40.296068 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.296026    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/69881682-7640-4077-8ff4-836f62457440-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:40.296222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.296083    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/69881682-7640-4077-8ff4-836f62457440-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:40.296222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.296094    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlq5h\" (UniqueName: \"kubernetes.io/projected/69881682-7640-4077-8ff4-836f62457440-kube-api-access-mlq5h\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:25:40.916397 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.916362    2576 generic.go:356] "Generic (PLEG): container finished" podID="69881682-7640-4077-8ff4-836f62457440" containerID="783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7" exitCode=0
Sep 04 22:25:40.916580 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.916479    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"
Sep 04 22:25:40.918982 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.918958    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx" event={"ID":"69881682-7640-4077-8ff4-836f62457440","Type":"ContainerDied","Data":"783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7"}
Sep 04 22:25:40.919108 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.919003    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx" event={"ID":"69881682-7640-4077-8ff4-836f62457440","Type":"ContainerDied","Data":"b6b9e1d67461a31f65c41dc11b688bd6f85ca5d82439f671a50e000ad3fa1a6f"}
Sep 04 22:25:40.919108 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.919020    2576 scope.go:117] "RemoveContainer" containerID="783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7"
Sep 04 22:25:40.928466 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.928290    2576 scope.go:117] "RemoveContainer" containerID="783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7"
Sep 04 22:25:40.928712 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:25:40.928532    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7\": container with ID starting with 783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7 not found: ID does not exist" containerID="783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7"
Sep 04 22:25:40.928712 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.928554    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7"} err="failed to get container status \"783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7\": rpc error: code = NotFound desc = could not find container \"783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7\": container with ID starting with 783609abe8fb09d09c01adf4536a681e8ca5f8d34e673a9b61956748d31ebac7 not found: ID does not exist"
Sep 04 22:25:40.944594 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.944572    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"]
Sep 04 22:25:40.952204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:40.952181    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-a8d845/autores-model-a8d845-kserve-566c9598b7-4fpsx"]
Sep 04 22:25:42.919009 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:42.918971    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69881682-7640-4077-8ff4-836f62457440" path="/var/lib/kubelet/pods/69881682-7640-4077-8ff4-836f62457440/volumes"
Sep 04 22:25:43.212468 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:43.212421    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:43.212741 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:43.212705    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:43.225931 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:43.225904    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:43.943913 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:43.943885    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:25:48.902861 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:48.902826    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:25:57.178922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:57.178884    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:59.499470 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.499433    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"]
Sep 04 22:25:59.499872 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.499857    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69881682-7640-4077-8ff4-836f62457440" containerName="main"
Sep 04 22:25:59.499927 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.499875    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="69881682-7640-4077-8ff4-836f62457440" containerName="main"
Sep 04 22:25:59.499967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.499955    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="69881682-7640-4077-8ff4-836f62457440" containerName="main"
Sep 04 22:25:59.504349 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.504331    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.506959 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.506941    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-8b0d12be-kserve-self-signed-certs\""
Sep 04 22:25:59.513814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.513790    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"]
Sep 04 22:25:59.519253 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.519222    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:25:59.586343 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.586304    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-dshm\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.586555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.586373    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tmp-dir\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.586555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.586431    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tls-certs\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.586555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.586543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-home\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.586739 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.586594    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8rk8\" (UniqueName: \"kubernetes.io/projected/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-kube-api-access-c8rk8\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.586739 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.586637    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-model-cache\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.687907 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.687866    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tls-certs\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.688137 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.687968    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-home\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.688137 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.687994    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8rk8\" (UniqueName: \"kubernetes.io/projected/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-kube-api-access-c8rk8\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.688137 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.688025    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-model-cache\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.688137 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.688094    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-dshm\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.688342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.688149    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tmp-dir\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.688492 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.688463    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-home\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.688637 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.688501    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-model-cache\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.688637 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.688541    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tmp-dir\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.690228 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.690210    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-dshm\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.690484 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.690467    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tls-certs\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.699631 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.699612    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8rk8\" (UniqueName: \"kubernetes.io/projected/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-kube-api-access-c8rk8\") pod \"e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") " pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:25:59.815175 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:25:59.815095    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:26:00.096714 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:00.095409    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"]
Sep 04 22:26:00.097752 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:26:00.097724    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod91ae1c16_26fa_41bb_8e2d_2208b7dbe1ac.slice/crio-a031025ba086aa51fe31929b16a60ccfca0f588a7c8a2ab8caed05f6e549b338 WatchSource:0}: Error finding container a031025ba086aa51fe31929b16a60ccfca0f588a7c8a2ab8caed05f6e549b338: Status 404 returned error can't find the container with id a031025ba086aa51fe31929b16a60ccfca0f588a7c8a2ab8caed05f6e549b338
Sep 04 22:26:00.189046 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:00.189007    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:00.908364 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:00.908329    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:01.023967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:01.023928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd" event={"ID":"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac","Type":"ContainerStarted","Data":"b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901"}
Sep 04 22:26:01.024206 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:01.023978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd" event={"ID":"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac","Type":"ContainerStarted","Data":"a031025ba086aa51fe31929b16a60ccfca0f588a7c8a2ab8caed05f6e549b338"}
Sep 04 22:26:01.046140 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:01.046085    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd" podStartSLOduration=2.046067103 podStartE2EDuration="2.046067103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:25:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:26:01.044530285 +0000 UTC m=+1044.563252820" watchObservedRunningTime="2026-09-04 22:26:01.046067103 +0000 UTC m=+1044.564789626"
Sep 04 22:26:04.599369 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:04.599331    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:26:06.740808 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:06.740773    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:06.848631 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:06.848577    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:06.943264 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:06.943232    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:09.815529 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:09.815490    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:26:09.815951 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:09.815544    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:26:09.828110 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:09.828084    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:26:10.081674 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:10.081585    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:26:12.344563 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:12.344526    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:12.573919 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:12.573883    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"]
Sep 04 22:26:12.574208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:12.574180    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy" containerID="cri-o://0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987" gracePeriod=30
Sep 04 22:26:12.599627 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:12.599535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:12.660376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:12.660336    2576 patch_prober.go:28] interesting pod/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 22:26:12.660528 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:12.660405    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:26:13.044325 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:13.044285    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:13.351424 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:13.351331    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:13.446560 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:13.446525    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:16.672385 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:16.672283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:16.762370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:16.762312    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:17.790412 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.790377    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"]
Sep 04 22:26:17.795850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.795823    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.798437 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.798411    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-b91726-serving-cert\""
Sep 04 22:26:17.807220 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.807151    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"]
Sep 04 22:26:17.830390 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.830204    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:17.854108 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.854024    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:26:17.870864 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.870832    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-data\") pod \"14438d60-468e-49e2-a864-4e863cc84594\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") "
Sep 04 22:26:17.871082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.870872    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-envoy\") pod \"14438d60-468e-49e2-a864-4e863cc84594\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") "
Sep 04 22:26:17.871082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.870964    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14438d60-468e-49e2-a864-4e863cc84594-istiod-ca-cert\") pod \"14438d60-468e-49e2-a864-4e863cc84594\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") "
Sep 04 22:26:17.871082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.870989    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-istio-token\") pod \"14438d60-468e-49e2-a864-4e863cc84594\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") "
Sep 04 22:26:17.871082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871011    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsg6j\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-kube-api-access-zsg6j\") pod \"14438d60-468e-49e2-a864-4e863cc84594\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") "
Sep 04 22:26:17.871082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871047    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-credential-socket\") pod \"14438d60-468e-49e2-a864-4e863cc84594\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") "
Sep 04 22:26:17.871356 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871175    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-data" (OuterVolumeSpecName: "istio-data") pod "14438d60-468e-49e2-a864-4e863cc84594" (UID: "14438d60-468e-49e2-a864-4e863cc84594"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:26:17.871356 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871212    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-socket\") pod \"14438d60-468e-49e2-a864-4e863cc84594\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") "
Sep 04 22:26:17.871356 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871263    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14438d60-468e-49e2-a864-4e863cc84594-istio-podinfo\") pod \"14438d60-468e-49e2-a864-4e863cc84594\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") "
Sep 04 22:26:17.871356 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871310    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-certs\") pod \"14438d60-468e-49e2-a864-4e863cc84594\" (UID: \"14438d60-468e-49e2-a864-4e863cc84594\") "
Sep 04 22:26:17.871574 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871491    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14438d60-468e-49e2-a864-4e863cc84594-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "14438d60-468e-49e2-a864-4e863cc84594" (UID: "14438d60-468e-49e2-a864-4e863cc84594"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:26:17.871574 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871501    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "14438d60-468e-49e2-a864-4e863cc84594" (UID: "14438d60-468e-49e2-a864-4e863cc84594"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:26:17.871574 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871531    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-openshift-service-ca\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.871574 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871561    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b27zz\" (UniqueName: \"kubernetes.io/projected/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-kube-api-access-b27zz\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.871794 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871648    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-metrics-certs\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.871794 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871720    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "14438d60-468e-49e2-a864-4e863cc84594" (UID: "14438d60-468e-49e2-a864-4e863cc84594"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:26:17.871794 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871788    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-maas-api-tls\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.871966 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871946    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:17.871966 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871964    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-data\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:17.872085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871978    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14438d60-468e-49e2-a864-4e863cc84594-istiod-ca-cert\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:17.872085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.871993    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-workload-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:17.872085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.872045    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "14438d60-468e-49e2-a864-4e863cc84594" (UID: "14438d60-468e-49e2-a864-4e863cc84594"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:26:17.873370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.873344    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "14438d60-468e-49e2-a864-4e863cc84594" (UID: "14438d60-468e-49e2-a864-4e863cc84594"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:26:17.873474 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.873443    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-istio-token" (OuterVolumeSpecName: "istio-token") pod "14438d60-468e-49e2-a864-4e863cc84594" (UID: "14438d60-468e-49e2-a864-4e863cc84594"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:26:17.873643 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.873623    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14438d60-468e-49e2-a864-4e863cc84594-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "14438d60-468e-49e2-a864-4e863cc84594" (UID: "14438d60-468e-49e2-a864-4e863cc84594"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 22:26:17.873762 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.873742    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-kube-api-access-zsg6j" (OuterVolumeSpecName: "kube-api-access-zsg6j") pod "14438d60-468e-49e2-a864-4e863cc84594" (UID: "14438d60-468e-49e2-a864-4e863cc84594"). InnerVolumeSpecName "kube-api-access-zsg6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:26:17.904088 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.904034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:17.973349 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973309    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-openshift-service-ca\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.973349 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973347    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b27zz\" (UniqueName: \"kubernetes.io/projected/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-kube-api-access-b27zz\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.973615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-metrics-certs\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.973615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973416    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-maas-api-tls\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.973615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973454    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-istio-token\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:17.973615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973463    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsg6j\" (UniqueName: \"kubernetes.io/projected/14438d60-468e-49e2-a864-4e863cc84594-kube-api-access-zsg6j\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:17.973615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973474    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-credential-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:17.973615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973487    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14438d60-468e-49e2-a864-4e863cc84594-istio-podinfo\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:17.973615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973499    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14438d60-468e-49e2-a864-4e863cc84594-istio-envoy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:17.973949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.973927    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-openshift-service-ca\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.976024 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.975991    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-metrics-certs\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.976153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.976000    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-maas-api-tls\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:17.983505 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:17.983483    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b27zz\" (UniqueName: \"kubernetes.io/projected/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-kube-api-access-b27zz\") pod \"maas-api-e2e-explicit-b91726-549d7c9948-wl2d4\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:18.110438 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.110335    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:18.115390 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.115361    2576 generic.go:356] "Generic (PLEG): container finished" podID="14438d60-468e-49e2-a864-4e863cc84594" containerID="0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987" exitCode=0
Sep 04 22:26:18.115503 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.115409    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"
Sep 04 22:26:18.115503 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.115457    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" event={"ID":"14438d60-468e-49e2-a864-4e863cc84594","Type":"ContainerDied","Data":"0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987"}
Sep 04 22:26:18.115577 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.115508    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb" event={"ID":"14438d60-468e-49e2-a864-4e863cc84594","Type":"ContainerDied","Data":"0f7d4744533491348ef885f2fb473f930d6c03c7adbcbd13d21df69c4add05f1"}
Sep 04 22:26:18.115577 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.115532    2576 scope.go:117] "RemoveContainer" containerID="0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987"
Sep 04 22:26:18.125136 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.125117    2576 scope.go:117] "RemoveContainer" containerID="ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5"
Sep 04 22:26:18.133516 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.133497    2576 scope.go:117] "RemoveContainer" containerID="0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987"
Sep 04 22:26:18.133834 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:26:18.133812    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987\": container with ID starting with 0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987 not found: ID does not exist" containerID="0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987"
Sep 04 22:26:18.133924 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.133841    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987"} err="failed to get container status \"0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987\": rpc error: code = NotFound desc = could not find container \"0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987\": container with ID starting with 0e0b771b878666a7aa716493f5b4721a81103e914eb68cedad14ab9ce246d987 not found: ID does not exist"
Sep 04 22:26:18.133924 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.133859    2576 scope.go:117] "RemoveContainer" containerID="ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5"
Sep 04 22:26:18.134151 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:26:18.134129    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5\": container with ID starting with ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5 not found: ID does not exist" containerID="ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5"
Sep 04 22:26:18.134225 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.134157    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5"} err="failed to get container status \"ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5\": rpc error: code = NotFound desc = could not find container \"ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5\": container with ID starting with ea9a17cf6300930f8a310e2809bd54de658357cff433072982d2b117670bafe5 not found: ID does not exist"
Sep 04 22:26:18.138877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.138855    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"]
Sep 04 22:26:18.145043 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.145022    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-a8d845-openshift-default-8696b856d6-b96tb"]
Sep 04 22:26:18.346601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.346568    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:18.386799 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.386775    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"]
Sep 04 22:26:18.389669 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:26:18.389640    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd304fa11_4d3d_4f6f_bbc2_ffa40be73dc5.slice/crio-1471e9e27d05fbcc4737d3d512e50e12b1f706ce28a61209dfae25f78b5eb7af WatchSource:0}: Error finding container 1471e9e27d05fbcc4737d3d512e50e12b1f706ce28a61209dfae25f78b5eb7af: Status 404 returned error can't find the container with id 1471e9e27d05fbcc4737d3d512e50e12b1f706ce28a61209dfae25f78b5eb7af
Sep 04 22:26:18.446233 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.446201    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:18.919074 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:18.919021    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14438d60-468e-49e2-a864-4e863cc84594" path="/var/lib/kubelet/pods/14438d60-468e-49e2-a864-4e863cc84594/volumes"
Sep 04 22:26:19.122559 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:19.122447    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4" event={"ID":"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5","Type":"ContainerStarted","Data":"c79f1b809a514bcb583e28cf9b6ca90fa96e5398edac465c1a4397ccc6e366cc"}
Sep 04 22:26:19.122559 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:19.122527    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:19.122559 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:19.122554    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4" event={"ID":"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5","Type":"ContainerStarted","Data":"1471e9e27d05fbcc4737d3d512e50e12b1f706ce28a61209dfae25f78b5eb7af"}
Sep 04 22:26:19.142635 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:19.142583    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4" podStartSLOduration=1.683138225 podStartE2EDuration="2.142568751s" totalImagesPullingTime="459.430526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:26:19.140761297 +0000 UTC m=+1062.659484069" watchObservedRunningTime="2026-09-04 22:26:19.142568751 +0000 UTC m=+1062.661291283"
Sep 04 22:26:22.277178 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:22.277141    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"]
Sep 04 22:26:22.301408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:22.301365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:22.803716 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:22.803682    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:25.135891 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:25.135862    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:26:26.104134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:26.104101    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:26:28.986643 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:28.986563    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:29.364829 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:29.364743    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:29.460872 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:29.460840    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:37.297567 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.297454    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd" podUID="91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" containerName="main" containerID="cri-o://b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901" gracePeriod=45
Sep 04 22:26:37.300300 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.300271    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"]
Sep 04 22:26:37.300783 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.300763    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy"
Sep 04 22:26:37.300935 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.300852    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy"
Sep 04 22:26:37.300935 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.300878    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy"
Sep 04 22:26:37.300935 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.300887    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy"
Sep 04 22:26:37.301123 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.301014    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy"
Sep 04 22:26:37.301123 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.301028    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="14438d60-468e-49e2-a864-4e863cc84594" containerName="istio-proxy"
Sep 04 22:26:37.305381 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.305362    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.309178 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.309155    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fecd36\"/\"default-dockercfg-42x6s\""
Sep 04 22:26:37.309280 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.309186    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fecd36\"/\"test-model-fecd36-kserve-self-signed-certs\""
Sep 04 22:26:37.309280 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.309218    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fecd36\"/\"openshift-service-ca.crt\""
Sep 04 22:26:37.309280 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.309255    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fecd36\"/\"kube-root-ca.crt\""
Sep 04 22:26:37.313155 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.313133    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"]
Sep 04 22:26:37.359339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.359306    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:37.467754 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.467723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-dshm\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.467895 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.467775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-home\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.467895 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.467823    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tls-certs\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.467970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.467925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tmp-dir\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.467970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.467951    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-model-cache\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.468039 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.467985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fm2g9\" (UniqueName: \"kubernetes.io/projected/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-kube-api-access-fm2g9\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.547735 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.547663    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:26:37.569150 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.569110    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fm2g9\" (UniqueName: \"kubernetes.io/projected/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-kube-api-access-fm2g9\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.569330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.569189    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-dshm\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.569330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.569244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-home\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.569330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.569284    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tls-certs\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.569494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.569398    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tmp-dir\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.569494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.569427    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-model-cache\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.569893 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.569865    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-model-cache\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.570240 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.570193    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-home\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.570370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.570250    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tmp-dir\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.572247 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.572224    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tls-certs\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.572352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.572310    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-dshm\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.582417 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.582386    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fm2g9\" (UniqueName: \"kubernetes.io/projected/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-kube-api-access-fm2g9\") pod \"test-model-fecd36-kserve-77cb99db55-ckdbt\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") " pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.670234 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670199    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tmp-dir\") pod \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") "
Sep 04 22:26:37.670408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670243    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-model-cache\") pod \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") "
Sep 04 22:26:37.670408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670274    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8rk8\" (UniqueName: \"kubernetes.io/projected/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-kube-api-access-c8rk8\") pod \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") "
Sep 04 22:26:37.670408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670335    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-dshm\") pod \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") "
Sep 04 22:26:37.670408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670397    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tls-certs\") pod \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") "
Sep 04 22:26:37.670613 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670425    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-home\") pod \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\" (UID: \"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac\") "
Sep 04 22:26:37.670613 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670490    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-model-cache" (OuterVolumeSpecName: "model-cache") pod "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" (UID: "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:26:37.670613 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670508    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" (UID: "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:26:37.670791 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670743    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-home" (OuterVolumeSpecName: "home") pod "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" (UID: "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:26:37.670833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670809    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:37.670833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.670828    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:37.672569 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.672544    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-dshm" (OuterVolumeSpecName: "dshm") pod "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" (UID: "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:26:37.672949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.672927    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" (UID: "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:26:37.672998 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.672942    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-kube-api-access-c8rk8" (OuterVolumeSpecName: "kube-api-access-c8rk8") pod "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" (UID: "91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac"). InnerVolumeSpecName "kube-api-access-c8rk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:26:37.680118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.680101    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:37.771443 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.771409    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8rk8\" (UniqueName: \"kubernetes.io/projected/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-kube-api-access-c8rk8\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:37.771443 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.771437    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:37.771443 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.771447    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:37.771662 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.771456    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:26:37.952867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:37.952843    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"]
Sep 04 22:26:37.954739 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:26:37.954715    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podab8ecc75_0aeb_47d7_bd45_7077ab1a7f3b.slice/crio-7660ac8365b6479886056d6bd2a505c68bb702b31bd63008bc1fc56bbc3ba52b WatchSource:0}: Error finding container 7660ac8365b6479886056d6bd2a505c68bb702b31bd63008bc1fc56bbc3ba52b: Status 404 returned error can't find the container with id 7660ac8365b6479886056d6bd2a505c68bb702b31bd63008bc1fc56bbc3ba52b
Sep 04 22:26:38.229250 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.229212    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt" event={"ID":"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b","Type":"ContainerStarted","Data":"aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7"}
Sep 04 22:26:38.229250 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.229255    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt" event={"ID":"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b","Type":"ContainerStarted","Data":"7660ac8365b6479886056d6bd2a505c68bb702b31bd63008bc1fc56bbc3ba52b"}
Sep 04 22:26:38.230520 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.230495    2576 generic.go:356] "Generic (PLEG): container finished" podID="91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" containerID="b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901" exitCode=0
Sep 04 22:26:38.230667 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.230528    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"
Sep 04 22:26:38.230667 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.230576    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd" event={"ID":"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac","Type":"ContainerDied","Data":"b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901"}
Sep 04 22:26:38.230667 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.230614    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd" event={"ID":"91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac","Type":"ContainerDied","Data":"a031025ba086aa51fe31929b16a60ccfca0f588a7c8a2ab8caed05f6e549b338"}
Sep 04 22:26:38.230667 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.230633    2576 scope.go:117] "RemoveContainer" containerID="b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901"
Sep 04 22:26:38.240033 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.240016    2576 scope.go:117] "RemoveContainer" containerID="b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901"
Sep 04 22:26:38.240360 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:26:38.240336    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901\": container with ID starting with b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901 not found: ID does not exist" containerID="b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901"
Sep 04 22:26:38.240413 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.240370    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901"} err="failed to get container status \"b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901\": rpc error: code = NotFound desc = could not find container \"b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901\": container with ID starting with b74aabe318201d56455ccebb5cbecbcb55678b60e131fcd8192e99233dd5d901 not found: ID does not exist"
Sep 04 22:26:38.247546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.247502    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt" podStartSLOduration=1.247488402 podStartE2EDuration="1.247488402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:26:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:26:38.246677717 +0000 UTC m=+1081.765400229" watchObservedRunningTime="2026-09-04 22:26:38.247488402 +0000 UTC m=+1081.766210935"
Sep 04 22:26:38.261034 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.261002    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"]
Sep 04 22:26:38.265083 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.265044    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-8b0d12be-kserve-6bd98dd579-slnpd"]
Sep 04 22:26:38.919108 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:38.919074    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" path="/var/lib/kubelet/pods/91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac/volumes"
Sep 04 22:26:39.731437 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:39.731408    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:40.405247 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:40.405209    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:40.538807 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:40.538772    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:41.619104 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:41.619040    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:46.795493 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:46.795451    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:46.899347 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:46.899311    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:47.011490 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:47.011457    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:47.086191 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:47.086106    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:26:47.260839 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:47.260800    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:47.681087 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:47.681030    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:47.681265 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:47.681106    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:47.693946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:47.693909    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:47.831149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:47.831103    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:47.926492 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:47.926458    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:48.027308 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:48.027261    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:48.299800 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:48.299719    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:26:49.454671 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:49.454627    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:55.382481 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:55.382445    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:55.451831 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:55.451795    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:26:56.080476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:26:56.080441    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:02.515473 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:02.515436    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:02.582357 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:02.582312    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:02.696415 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:02.696380    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:27:02.749136 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:02.749097    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:03.093878 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:03.093836    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:03.174440 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:03.174404    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:05.984189 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:05.984150    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:06.075239 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:06.075201    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:07.103932 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.103899    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"]
Sep 04 22:27:07.104551 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.104528    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" containerName="main"
Sep 04 22:27:07.104624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.104553    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" containerName="main"
Sep 04 22:27:07.104681 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.104674    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ae1c16-26fa-41bb-8e2d-2208b7dbe1ac" containerName="main"
Sep 04 22:27:07.107588 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.107568    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.110070 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.110033    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-84624e97-serving-cert\""
Sep 04 22:27:07.120091 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.120044    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"]
Sep 04 22:27:07.144824 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.144786    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:07.151046 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.151016    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-maas-api-tls\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.151204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.151073    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vfg5\" (UniqueName: \"kubernetes.io/projected/ae79be16-d4c9-41c4-8faf-1f97b49f13be-kube-api-access-6vfg5\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.151204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.151132    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79be16-d4c9-41c4-8faf-1f97b49f13be-openshift-service-ca\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.151204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.151179    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-metrics-certs\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.249156 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.249124    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:07.258878 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.258836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79be16-d4c9-41c4-8faf-1f97b49f13be-openshift-service-ca\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.259081 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.258932    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-metrics-certs\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.259163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.259138    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-maas-api-tls\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.259227 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.259209    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vfg5\" (UniqueName: \"kubernetes.io/projected/ae79be16-d4c9-41c4-8faf-1f97b49f13be-kube-api-access-6vfg5\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.259694 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.259667    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79be16-d4c9-41c4-8faf-1f97b49f13be-openshift-service-ca\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.261660 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.261632    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-metrics-certs\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.261769 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.261644    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-maas-api-tls\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.269458 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.269431    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vfg5\" (UniqueName: \"kubernetes.io/projected/ae79be16-d4c9-41c4-8faf-1f97b49f13be-kube-api-access-6vfg5\") pod \"maas-api-e2e-mt-84624e97-545bffc859-mcdf7\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.319579 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.319540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:07.420069 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.419950    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:07.672578 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.672473    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:07.693046 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.693020    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"]
Sep 04 22:27:07.696509 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:27:07.696481    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae79be16_d4c9_41c4_8faf_1f97b49f13be.slice/crio-31ba82b562264ab83f45de161f4694b7900af50b9b478b7f3e2d51d54b34e339 WatchSource:0}: Error finding container 31ba82b562264ab83f45de161f4694b7900af50b9b478b7f3e2d51d54b34e339: Status 404 returned error can't find the container with id 31ba82b562264ab83f45de161f4694b7900af50b9b478b7f3e2d51d54b34e339
Sep 04 22:27:07.698221 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.698200    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 22:27:07.767988 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:07.767949    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:08.391420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:08.391331    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7" event={"ID":"ae79be16-d4c9-41c4-8faf-1f97b49f13be","Type":"ContainerStarted","Data":"25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0"}
Sep 04 22:27:08.391420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:08.391398    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7" event={"ID":"ae79be16-d4c9-41c4-8faf-1f97b49f13be","Type":"ContainerStarted","Data":"31ba82b562264ab83f45de161f4694b7900af50b9b478b7f3e2d51d54b34e339"}
Sep 04 22:27:08.391810 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:08.391429    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:08.411127 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:08.411065    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7" podStartSLOduration=1.070661658 podStartE2EDuration="1.411033834s" totalImagesPullingTime="340.372176ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:27:08.409043765 +0000 UTC m=+1111.927766298" watchObservedRunningTime="2026-09-04 22:27:08.411033834 +0000 UTC m=+1111.929756366"
Sep 04 22:27:14.402234 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:14.402203    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:26.798267 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:26.798233    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:27:30.957420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:30.957381    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:31.624740 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:31.624701    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:31.721422 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:31.721386    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:32.065486 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:32.065456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:34.016882 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:34.016844    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:38.395706 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:38.395666    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:38.493212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:38.493172    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:40.709241 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:40.709203    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:44.507701 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:44.507663    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:44.837081 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:44.836964    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:48.903825 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:48.903787    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:49.100623 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.100583    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:27:49.638128 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.638095    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"]
Sep 04 22:27:49.638373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.638344    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7" podUID="ae79be16-d4c9-41c4-8faf-1f97b49f13be" containerName="maas-api" containerID="cri-o://25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0" gracePeriod=30
Sep 04 22:27:49.669387 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.669358    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:49.748069 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.748016    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:49.886582 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.886556    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:49.941549 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.941456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:49.973374 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.973344    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6vfg5\" (UniqueName: \"kubernetes.io/projected/ae79be16-d4c9-41c4-8faf-1f97b49f13be-kube-api-access-6vfg5\") pod \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") "
Sep 04 22:27:49.973528 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.973382    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-maas-api-tls\") pod \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") "
Sep 04 22:27:49.973528 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.973444    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-metrics-certs\") pod \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") "
Sep 04 22:27:49.973528 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.973472    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79be16-d4c9-41c4-8faf-1f97b49f13be-openshift-service-ca\") pod \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\" (UID: \"ae79be16-d4c9-41c4-8faf-1f97b49f13be\") "
Sep 04 22:27:49.973916 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.973890    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae79be16-d4c9-41c4-8faf-1f97b49f13be-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ae79be16-d4c9-41c4-8faf-1f97b49f13be" (UID: "ae79be16-d4c9-41c4-8faf-1f97b49f13be"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:27:49.975591 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.975564    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae79be16-d4c9-41c4-8faf-1f97b49f13be-kube-api-access-6vfg5" (OuterVolumeSpecName: "kube-api-access-6vfg5") pod "ae79be16-d4c9-41c4-8faf-1f97b49f13be" (UID: "ae79be16-d4c9-41c4-8faf-1f97b49f13be"). InnerVolumeSpecName "kube-api-access-6vfg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:27:49.975705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.975670    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ae79be16-d4c9-41c4-8faf-1f97b49f13be" (UID: "ae79be16-d4c9-41c4-8faf-1f97b49f13be"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:27:49.975705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:49.975681    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ae79be16-d4c9-41c4-8faf-1f97b49f13be" (UID: "ae79be16-d4c9-41c4-8faf-1f97b49f13be"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:27:50.014152 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.014101    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:50.075142 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.075103    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:27:50.075142 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.075138    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae79be16-d4c9-41c4-8faf-1f97b49f13be-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:27:50.075142 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.075150    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vfg5\" (UniqueName: \"kubernetes.io/projected/ae79be16-d4c9-41c4-8faf-1f97b49f13be-kube-api-access-6vfg5\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:27:50.075390 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.075162    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ae79be16-d4c9-41c4-8faf-1f97b49f13be-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:27:50.262689 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.262646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:50.607731 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.607640    2576 generic.go:356] "Generic (PLEG): container finished" podID="ae79be16-d4c9-41c4-8faf-1f97b49f13be" containerID="25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0" exitCode=0
Sep 04 22:27:50.607731 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.607679    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"
Sep 04 22:27:50.607973 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.607733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7" event={"ID":"ae79be16-d4c9-41c4-8faf-1f97b49f13be","Type":"ContainerDied","Data":"25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0"}
Sep 04 22:27:50.607973 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.607789    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7" event={"ID":"ae79be16-d4c9-41c4-8faf-1f97b49f13be","Type":"ContainerDied","Data":"31ba82b562264ab83f45de161f4694b7900af50b9b478b7f3e2d51d54b34e339"}
Sep 04 22:27:50.607973 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.607819    2576 scope.go:117] "RemoveContainer" containerID="25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0"
Sep 04 22:27:50.617693 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.617676    2576 scope.go:117] "RemoveContainer" containerID="25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0"
Sep 04 22:27:50.617959 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:27:50.617940    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0\": container with ID starting with 25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0 not found: ID does not exist" containerID="25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0"
Sep 04 22:27:50.618026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.617965    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0"} err="failed to get container status \"25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0\": rpc error: code = NotFound desc = could not find container \"25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0\": container with ID starting with 25279f7e99c235876bcf7f9cf9b54e96a5c06f337db1916e63012a7d3c0c1ef0 not found: ID does not exist"
Sep 04 22:27:50.630933 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.630908    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"]
Sep 04 22:27:50.636042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.636006    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-84624e97-545bffc859-mcdf7"]
Sep 04 22:27:50.918518 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:50.918434    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae79be16-d4c9-41c4-8faf-1f97b49f13be" path="/var/lib/kubelet/pods/ae79be16-d4c9-41c4-8faf-1f97b49f13be/volumes"
Sep 04 22:27:54.746361 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:54.746310    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:55.443406 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.443369    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"]
Sep 04 22:27:55.443618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.443594    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4" podUID="d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" containerName="maas-api" containerID="cri-o://c79f1b809a514bcb583e28cf9b6ca90fa96e5398edac465c1a4397ccc6e366cc" gracePeriod=30
Sep 04 22:27:55.470795 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.470764    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:55.545376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.545347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:55.637634 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.637600    2576 generic.go:356] "Generic (PLEG): container finished" podID="d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" containerID="c79f1b809a514bcb583e28cf9b6ca90fa96e5398edac465c1a4397ccc6e366cc" exitCode=0
Sep 04 22:27:55.637802 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.637671    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4" event={"ID":"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5","Type":"ContainerDied","Data":"c79f1b809a514bcb583e28cf9b6ca90fa96e5398edac465c1a4397ccc6e366cc"}
Sep 04 22:27:55.645016 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.644988    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:55.691892 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.691866    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:27:55.713115 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.713002    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:55.832928 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.832897    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-maas-api-tls\") pod \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") "
Sep 04 22:27:55.833414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.833067    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b27zz\" (UniqueName: \"kubernetes.io/projected/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-kube-api-access-b27zz\") pod \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") "
Sep 04 22:27:55.833414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.833106    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-openshift-service-ca\") pod \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") "
Sep 04 22:27:55.833414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.833146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-metrics-certs\") pod \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\" (UID: \"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5\") "
Sep 04 22:27:55.833611 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.833472    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" (UID: "d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:27:55.835285 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.835257    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-kube-api-access-b27zz" (OuterVolumeSpecName: "kube-api-access-b27zz") pod "d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" (UID: "d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5"). InnerVolumeSpecName "kube-api-access-b27zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:27:55.835571 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.835544    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" (UID: "d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:27:55.835692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.835612    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" (UID: "d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:27:55.934518 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.934477    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b27zz\" (UniqueName: \"kubernetes.io/projected/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-kube-api-access-b27zz\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:27:55.934518 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.934513    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:27:55.934518 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.934525    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:27:55.934765 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:55.934533    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:27:56.067355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:56.067270    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:56.644856 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:56.644822    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"
Sep 04 22:27:56.645112 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:56.644858    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4" event={"ID":"d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5","Type":"ContainerDied","Data":"1471e9e27d05fbcc4737d3d512e50e12b1f706ce28a61209dfae25f78b5eb7af"}
Sep 04 22:27:56.645112 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:56.644923    2576 scope.go:117] "RemoveContainer" containerID="c79f1b809a514bcb583e28cf9b6ca90fa96e5398edac465c1a4397ccc6e366cc"
Sep 04 22:27:56.671097 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:56.670717    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"]
Sep 04 22:27:56.675991 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:56.675964    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-b91726-549d7c9948-wl2d4"]
Sep 04 22:27:56.918945 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:56.918871    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" path="/var/lib/kubelet/pods/d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5/volumes"
Sep 04 22:27:57.127358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:57.127315    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:57.227949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:57.227912    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:57.409957 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:57.409921    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:57.801178 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:57.801148    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:57.902249 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:57.902215    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:27:57.999307 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:27:57.999254    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:01.145089 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:01.144958    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:03.278262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:03.278226    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"]
Sep 04 22:28:03.326853 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:03.326815    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:04.284644 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:04.284610    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"]
Sep 04 22:28:04.310343 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:04.310310    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:04.540320 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:04.540232    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:04.666072 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:04.666017    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:04.781463 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:04.781427    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:04.871035 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:04.870949    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:05.014221 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:05.014183    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:06.799710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:06.799673    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:28:07.883739 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:07.883695    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:08.108443 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.108413    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:08.572408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.572366    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:08.837179 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.837085    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"]
Sep 04 22:28:08.837567 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.837551    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae79be16-d4c9-41c4-8faf-1f97b49f13be" containerName="maas-api"
Sep 04 22:28:08.837634 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.837567    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae79be16-d4c9-41c4-8faf-1f97b49f13be" containerName="maas-api"
Sep 04 22:28:08.837634 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.837597    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" containerName="maas-api"
Sep 04 22:28:08.837634 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.837603    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" containerName="maas-api"
Sep 04 22:28:08.837785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.837676    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="d304fa11-4d3d-4f6f-bbc2-ffa40be73dc5" containerName="maas-api"
Sep 04 22:28:08.837785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.837684    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae79be16-d4c9-41c4-8faf-1f97b49f13be" containerName="maas-api"
Sep 04 22:28:08.841026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.841008    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.843689 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.843668    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-e64d5c-openshift-default-dockercfg-gx6qf\""
Sep 04 22:28:08.851205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.851176    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"]
Sep 04 22:28:08.857946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.857911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:08.860889 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.860866    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istiod-ca-cert\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.861005 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.860941    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-podinfo\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.861005 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.860968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-envoy\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.861005 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.860998    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz42k\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-kube-api-access-hz42k\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.861205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.861021    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-token\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.861205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.861118    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-socket\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.861205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.861141    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-credential-socket\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.861205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.861165    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-certs\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.861416 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.861224    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-data\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.962700 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.962666    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istiod-ca-cert\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.962726    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-podinfo\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.962753    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-envoy\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.962778    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz42k\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-kube-api-access-hz42k\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.962806    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-token\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963543 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.963513    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istiod-ca-cert\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963543 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.963520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-socket\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.963573    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-credential-socket\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.963617    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-certs\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.963661    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-data\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963896 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.963775    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-socket\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.963963 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.963895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-credential-socket\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.964031 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.963963    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-certs\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.964119 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.964087    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-data\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.965706 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.965674    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-envoy\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.965806 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.965780    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-podinfo\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.972212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.972191    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-token\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:08.972610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:08.972593    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz42k\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-kube-api-access-hz42k\") pod \"e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") " pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:09.099686 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:09.099577    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:09.152136 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:09.152104    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:09.429103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:09.428975    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"]
Sep 04 22:28:09.432027 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:28:09.432000    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bfc06b_7aec_48a8_9c9b_d5193aefd5c6.slice/crio-6ee44acb78e6ff3f5cdac819b494440499b26da53c5be55060e8dcb500d300c7 WatchSource:0}: Error finding container 6ee44acb78e6ff3f5cdac819b494440499b26da53c5be55060e8dcb500d300c7: Status 404 returned error can't find the container with id 6ee44acb78e6ff3f5cdac819b494440499b26da53c5be55060e8dcb500d300c7
Sep 04 22:28:09.434680 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:09.434637    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:28:09.434764 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:09.434715    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:28:09.434764 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:09.434742    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:28:09.711792 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:09.711758    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq" event={"ID":"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6","Type":"ContainerStarted","Data":"144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631"}
Sep 04 22:28:09.711972 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:09.711808    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq" event={"ID":"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6","Type":"ContainerStarted","Data":"6ee44acb78e6ff3f5cdac819b494440499b26da53c5be55060e8dcb500d300c7"}
Sep 04 22:28:09.734676 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:09.734620    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq" podStartSLOduration=1.734603524 podStartE2EDuration="1.734603524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:09.732410405 +0000 UTC m=+1173.251132940" watchObservedRunningTime="2026-09-04 22:28:09.734603524 +0000 UTC m=+1173.253326060"
Sep 04 22:28:10.098460 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:10.098365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:10.152247 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:10.152213    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:10.157118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:10.157092    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:10.183814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:10.183775    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:10.280046 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:10.280009    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:10.716569 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:10.716538    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:10.717377 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:10.717352    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:12.258818 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:12.258789    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:12.352415 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:12.352384    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:13.213244 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:13.213210    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:13.325443 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:13.325403    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:13.494168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:13.494034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:13.757143 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:13.757027    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:14.975109 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:14.975069    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:15.055847 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:15.055811    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:15.124398 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:15.124362    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:15.212875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:15.212832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:15.276383 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:15.276281    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:15.743741 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:15.743707    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:16.382373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:16.382339    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:16.464647 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:16.464610    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:17.744929 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:17.744896    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:17.853337 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:17.853301    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:18.139991 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.139904    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:18.222043 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.221914    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"]
Sep 04 22:28:18.222295 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.222158    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq" podUID="24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" containerName="istio-proxy" containerID="cri-o://144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631" gracePeriod=30
Sep 04 22:28:18.245803 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.245746    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:18.298500 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.298461    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x" podUID="5f6b9776-f870-421e-afc7-50a71cd347f4" containerName="main" containerID="cri-o://c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365" gracePeriod=45
Sep 04 22:28:18.557539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.557515    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:28:18.669338 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669262    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-26kbr\" (UniqueName: \"kubernetes.io/projected/5f6b9776-f870-421e-afc7-50a71cd347f4-kube-api-access-26kbr\") pod \"5f6b9776-f870-421e-afc7-50a71cd347f4\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") "
Sep 04 22:28:18.669338 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669309    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-dshm\") pod \"5f6b9776-f870-421e-afc7-50a71cd347f4\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") "
Sep 04 22:28:18.669535 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669384    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-model-cache\") pod \"5f6b9776-f870-421e-afc7-50a71cd347f4\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") "
Sep 04 22:28:18.669535 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669506    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6b9776-f870-421e-afc7-50a71cd347f4-tls-certs\") pod \"5f6b9776-f870-421e-afc7-50a71cd347f4\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") "
Sep 04 22:28:18.669646 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669557    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-home\") pod \"5f6b9776-f870-421e-afc7-50a71cd347f4\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") "
Sep 04 22:28:18.669646 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669599    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-tmp-dir\") pod \"5f6b9776-f870-421e-afc7-50a71cd347f4\" (UID: \"5f6b9776-f870-421e-afc7-50a71cd347f4\") "
Sep 04 22:28:18.669738 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669672    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-model-cache" (OuterVolumeSpecName: "model-cache") pod "5f6b9776-f870-421e-afc7-50a71cd347f4" (UID: "5f6b9776-f870-421e-afc7-50a71cd347f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:18.669879 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669831    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-home" (OuterVolumeSpecName: "home") pod "5f6b9776-f870-421e-afc7-50a71cd347f4" (UID: "5f6b9776-f870-421e-afc7-50a71cd347f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:18.670007 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669945    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5f6b9776-f870-421e-afc7-50a71cd347f4" (UID: "5f6b9776-f870-421e-afc7-50a71cd347f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:18.670007 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.669999    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:18.670166 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.670014    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:18.671606 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.671578    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:18.671982 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.671957    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-dshm" (OuterVolumeSpecName: "dshm") pod "5f6b9776-f870-421e-afc7-50a71cd347f4" (UID: "5f6b9776-f870-421e-afc7-50a71cd347f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:18.672104 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.672021    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f6b9776-f870-421e-afc7-50a71cd347f4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5f6b9776-f870-421e-afc7-50a71cd347f4" (UID: "5f6b9776-f870-421e-afc7-50a71cd347f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:28:18.672104 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.672077    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f6b9776-f870-421e-afc7-50a71cd347f4-kube-api-access-26kbr" (OuterVolumeSpecName: "kube-api-access-26kbr") pod "5f6b9776-f870-421e-afc7-50a71cd347f4" (UID: "5f6b9776-f870-421e-afc7-50a71cd347f4"). InnerVolumeSpecName "kube-api-access-26kbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:28:18.755994 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.755966    2576 generic.go:356] "Generic (PLEG): container finished" podID="5f6b9776-f870-421e-afc7-50a71cd347f4" containerID="c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365" exitCode=0
Sep 04 22:28:18.756376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.756008    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x" event={"ID":"5f6b9776-f870-421e-afc7-50a71cd347f4","Type":"ContainerDied","Data":"c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365"}
Sep 04 22:28:18.756376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.756011    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"
Sep 04 22:28:18.756376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.756044    2576 scope.go:117] "RemoveContainer" containerID="c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365"
Sep 04 22:28:18.756376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.756035    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x" event={"ID":"5f6b9776-f870-421e-afc7-50a71cd347f4","Type":"ContainerDied","Data":"775dff686c80c6e071aa12a2e6e8b5698c69d39d4b4c2757ee0b0c014ecc08e2"}
Sep 04 22:28:18.764835 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.764819    2576 scope.go:117] "RemoveContainer" containerID="c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365"
Sep 04 22:28:18.765114 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:28:18.765094    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365\": container with ID starting with c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365 not found: ID does not exist" containerID="c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365"
Sep 04 22:28:18.765201 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.765121    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365"} err="failed to get container status \"c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365\": rpc error: code = NotFound desc = could not find container \"c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365\": container with ID starting with c83d35ac9b0b8cd0069c9db5a7c2dc2ee7a1f0d6a7cbdbd288428fb2f9848365 not found: ID does not exist"
Sep 04 22:28:18.768097 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.768072    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:18.770566 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.770548    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:18.770639 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.770567    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f6b9776-f870-421e-afc7-50a71cd347f4-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:18.770639 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.770576    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f6b9776-f870-421e-afc7-50a71cd347f4-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:18.770639 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.770585    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26kbr\" (UniqueName: \"kubernetes.io/projected/5f6b9776-f870-421e-afc7-50a71cd347f4-kube-api-access-26kbr\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:18.778893 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.778871    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"]
Sep 04 22:28:18.785659 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.785640    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-d9f7ae/test-model-d9f7ae-kserve-6ddbbbdb89-2vz8x"]
Sep 04 22:28:18.924432 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:18.924342    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f6b9776-f870-421e-afc7-50a71cd347f4" path="/var/lib/kubelet/pods/5f6b9776-f870-421e-afc7-50a71cd347f4/volumes"
Sep 04 22:28:19.304132 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.304089    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt" podUID="ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" containerName="main" containerID="cri-o://aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7" gracePeriod=45
Sep 04 22:28:19.558682 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.558607    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:28:19.677910 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.677878    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-home\") pod \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") "
Sep 04 22:28:19.678102 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.677921    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tls-certs\") pod \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") "
Sep 04 22:28:19.678102 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.677937    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tmp-dir\") pod \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") "
Sep 04 22:28:19.678102 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.677974    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-dshm\") pod \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") "
Sep 04 22:28:19.678102 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.678010    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-model-cache\") pod \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") "
Sep 04 22:28:19.678334 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.678159    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fm2g9\" (UniqueName: \"kubernetes.io/projected/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-kube-api-access-fm2g9\") pod \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\" (UID: \"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b\") "
Sep 04 22:28:19.678393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.678335    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-home" (OuterVolumeSpecName: "home") pod "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" (UID: "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:19.678393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.678349    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" (UID: "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:19.678467 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.678381    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" (UID: "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:19.678523 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.678506    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:19.678561 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.678541    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:19.678561 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.678551    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:19.680407 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.680385    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" (UID: "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:28:19.680498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.680386    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-dshm" (OuterVolumeSpecName: "dshm") pod "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" (UID: "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:19.680552 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.680527    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-kube-api-access-fm2g9" (OuterVolumeSpecName: "kube-api-access-fm2g9") pod "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" (UID: "ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b"). InnerVolumeSpecName "kube-api-access-fm2g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:28:19.707153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.707112    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:19.763205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.763164    2576 generic.go:356] "Generic (PLEG): container finished" podID="ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" containerID="aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7" exitCode=0
Sep 04 22:28:19.763645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.763222    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"
Sep 04 22:28:19.763645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.763259    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt" event={"ID":"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b","Type":"ContainerDied","Data":"aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7"}
Sep 04 22:28:19.763645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.763306    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt" event={"ID":"ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b","Type":"ContainerDied","Data":"7660ac8365b6479886056d6bd2a505c68bb702b31bd63008bc1fc56bbc3ba52b"}
Sep 04 22:28:19.763645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.763332    2576 scope.go:117] "RemoveContainer" containerID="aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7"
Sep 04 22:28:19.769868 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.769844    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:19.774846 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.774827    2576 scope.go:117] "RemoveContainer" containerID="aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7"
Sep 04 22:28:19.775234 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:28:19.775212    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7\": container with ID starting with aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7 not found: ID does not exist" containerID="aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7"
Sep 04 22:28:19.775331 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.775241    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7"} err="failed to get container status \"aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7\": rpc error: code = NotFound desc = could not find container \"aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7\": container with ID starting with aae12447a73a2590871cd72220663d843f5d7c779fc08bf8c2c119454a3127a7 not found: ID does not exist"
Sep 04 22:28:19.779565 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.779544    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:19.779565 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.779565    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:19.779687 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.779576    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fm2g9\" (UniqueName: \"kubernetes.io/projected/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b-kube-api-access-fm2g9\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:19.788552 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.788526    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"]
Sep 04 22:28:19.792476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:19.792454    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fecd36/test-model-fecd36-kserve-77cb99db55-ckdbt"]
Sep 04 22:28:20.401448 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:20.401413    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:28:20.918270 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:20.918237    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" path="/var/lib/kubelet/pods/ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b/volumes"
Sep 04 22:28:22.387049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:22.387005    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:22.460007 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:22.459973    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:22.641543 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:22.641449    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:23.471966 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.471944    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:23.617250 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617159    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz42k\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-kube-api-access-hz42k\") pod \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") "
Sep 04 22:28:23.617250 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617219    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-credential-socket\") pod \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") "
Sep 04 22:28:23.617497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617268    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-podinfo\") pod \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") "
Sep 04 22:28:23.617497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617302    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-socket\") pod \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") "
Sep 04 22:28:23.617497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617337    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-envoy\") pod \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") "
Sep 04 22:28:23.617497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617366    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-certs\") pod \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") "
Sep 04 22:28:23.617497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617403    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istiod-ca-cert\") pod \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") "
Sep 04 22:28:23.617497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617438    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-token\") pod \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") "
Sep 04 22:28:23.617787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617500    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-data\") pod \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\" (UID: \"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6\") "
Sep 04 22:28:23.617787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617568    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" (UID: "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:23.617787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.617782    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-credential-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:23.619314 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.619280    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" (UID: "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:23.619441 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.619390    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" (UID: "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:23.619441 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.619405    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-data" (OuterVolumeSpecName: "istio-data") pod "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" (UID: "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:23.619584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.619487    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" (UID: "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:28:23.619699 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.619680    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-kube-api-access-hz42k" (OuterVolumeSpecName: "kube-api-access-hz42k") pod "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" (UID: "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6"). InnerVolumeSpecName "kube-api-access-hz42k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:28:23.619799 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.619785    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" (UID: "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 22:28:23.621405 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.621379    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" (UID: "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:23.621405 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.621391    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-token" (OuterVolumeSpecName: "istio-token") pod "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" (UID: "24bfc06b-7aec-48a8-9c9b-d5193aefd5c6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:28:23.670802 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.670762    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:23.718919 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.718881    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:23.718919 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.718912    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istiod-ca-cert\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:23.718919 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.718921    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-token\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:23.718919 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.718930    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-data\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:23.719215 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.718939    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz42k\" (UniqueName: \"kubernetes.io/projected/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-kube-api-access-hz42k\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:23.719215 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.718951    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-podinfo\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:23.719215 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.718965    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-workload-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:23.719215 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.718974    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6-istio-envoy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:23.770751 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.770714    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:23.786224 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.786192    2576 generic.go:356] "Generic (PLEG): container finished" podID="24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" containerID="144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631" exitCode=0
Sep 04 22:28:23.786369 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.786241    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"
Sep 04 22:28:23.786369 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.786275    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq" event={"ID":"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6","Type":"ContainerDied","Data":"144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631"}
Sep 04 22:28:23.786369 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.786332    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq" event={"ID":"24bfc06b-7aec-48a8-9c9b-d5193aefd5c6","Type":"ContainerDied","Data":"6ee44acb78e6ff3f5cdac819b494440499b26da53c5be55060e8dcb500d300c7"}
Sep 04 22:28:23.786369 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.786356    2576 scope.go:117] "RemoveContainer" containerID="144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631"
Sep 04 22:28:23.795564 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.795547    2576 scope.go:117] "RemoveContainer" containerID="144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631"
Sep 04 22:28:23.795830 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:28:23.795809    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631\": container with ID starting with 144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631 not found: ID does not exist" containerID="144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631"
Sep 04 22:28:23.795875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.795839    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631"} err="failed to get container status \"144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631\": rpc error: code = NotFound desc = could not find container \"144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631\": container with ID starting with 144d7c66fee43fe415c518f4831e62fbaacc5be88717c6cab229c28f02720631 not found: ID does not exist"
Sep 04 22:28:23.809603 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.809580    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"]
Sep 04 22:28:23.814816 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:23.814794    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-e64d5c-openshift-default-8678f954b6-7n8fq"]
Sep 04 22:28:24.917829 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:24.917795    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" path="/var/lib/kubelet/pods/24bfc06b-7aec-48a8-9c9b-d5193aefd5c6/volumes"
Sep 04 22:28:33.691273 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:33.691226    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"]
Sep 04 22:28:33.691923 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:33.691454    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb" podUID="a43077ab-2bd6-4d17-a621-253842dcecac" containerName="maas-api" containerID="cri-o://3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98" gracePeriod=30
Sep 04 22:28:33.722880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:33.722849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:33.874637 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:33.874606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:33.957683 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:33.957649    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:34.020769 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.020734    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:34.145491 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.145468    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:28:34.325932 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.325849    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43077ab-2bd6-4d17-a621-253842dcecac-openshift-service-ca\") pod \"a43077ab-2bd6-4d17-a621-253842dcecac\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") "
Sep 04 22:28:34.325932 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.325903    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vhsd\" (UniqueName: \"kubernetes.io/projected/a43077ab-2bd6-4d17-a621-253842dcecac-kube-api-access-4vhsd\") pod \"a43077ab-2bd6-4d17-a621-253842dcecac\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") "
Sep 04 22:28:34.326211 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.325937    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-metrics-certs\") pod \"a43077ab-2bd6-4d17-a621-253842dcecac\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") "
Sep 04 22:28:34.326211 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.325960    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-maas-api-tls\") pod \"a43077ab-2bd6-4d17-a621-253842dcecac\" (UID: \"a43077ab-2bd6-4d17-a621-253842dcecac\") "
Sep 04 22:28:34.326327 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.326299    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43077ab-2bd6-4d17-a621-253842dcecac-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a43077ab-2bd6-4d17-a621-253842dcecac" (UID: "a43077ab-2bd6-4d17-a621-253842dcecac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:28:34.328110 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.328075    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a43077ab-2bd6-4d17-a621-253842dcecac" (UID: "a43077ab-2bd6-4d17-a621-253842dcecac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:28:34.328307 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.328109    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a43077ab-2bd6-4d17-a621-253842dcecac" (UID: "a43077ab-2bd6-4d17-a621-253842dcecac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:28:34.328307 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.328118    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43077ab-2bd6-4d17-a621-253842dcecac-kube-api-access-4vhsd" (OuterVolumeSpecName: "kube-api-access-4vhsd") pod "a43077ab-2bd6-4d17-a621-253842dcecac" (UID: "a43077ab-2bd6-4d17-a621-253842dcecac"). InnerVolumeSpecName "kube-api-access-4vhsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:28:34.361600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.361557    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:34.427813 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.427774    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vhsd\" (UniqueName: \"kubernetes.io/projected/a43077ab-2bd6-4d17-a621-253842dcecac-kube-api-access-4vhsd\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:34.427813 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.427807    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:34.427813 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.427817    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a43077ab-2bd6-4d17-a621-253842dcecac-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:34.428080 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.427825    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a43077ab-2bd6-4d17-a621-253842dcecac-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:34.844710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.844672    2576 generic.go:356] "Generic (PLEG): container finished" podID="a43077ab-2bd6-4d17-a621-253842dcecac" containerID="3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98" exitCode=0
Sep 04 22:28:34.844710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.844713    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"
Sep 04 22:28:34.845281 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.844757    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb" event={"ID":"a43077ab-2bd6-4d17-a621-253842dcecac","Type":"ContainerDied","Data":"3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98"}
Sep 04 22:28:34.845281 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.844810    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb" event={"ID":"a43077ab-2bd6-4d17-a621-253842dcecac","Type":"ContainerDied","Data":"e19f5c58d8c2b30dccbf82019e19001aad97f375d8b8fc5732312650a3c5b5c4"}
Sep 04 22:28:34.845281 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.844841    2576 scope.go:117] "RemoveContainer" containerID="3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98"
Sep 04 22:28:34.854231 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.854214    2576 scope.go:117] "RemoveContainer" containerID="3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98"
Sep 04 22:28:34.854484 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:28:34.854468    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98\": container with ID starting with 3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98 not found: ID does not exist" containerID="3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98"
Sep 04 22:28:34.854533 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.854492    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98"} err="failed to get container status \"3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98\": rpc error: code = NotFound desc = could not find container \"3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98\": container with ID starting with 3dae265f556ab9aa8b910f7dc529b08a18b804bf3a2345fc633aa653ef980a98 not found: ID does not exist"
Sep 04 22:28:34.867046 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.867022    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"]
Sep 04 22:28:34.871110 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.871089    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-fecd36-5c5549556b-cf6nb"]
Sep 04 22:28:34.918223 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:34.918189    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a43077ab-2bd6-4d17-a621-253842dcecac" path="/var/lib/kubelet/pods/a43077ab-2bd6-4d17-a621-253842dcecac/volumes"
Sep 04 22:28:36.947710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:36.947681    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:28:36.948525 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:36.948505    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:28:38.499763 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:38.499726    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:28:40.058319 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:40.058282    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:40.582146 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:40.582108    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:40.683775 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:40.683740    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:40.856928 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:40.856842    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"]
Sep 04 22:28:40.857214 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:40.857185    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy" containerID="cri-o://d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7" gracePeriod=30
Sep 04 22:28:40.904939 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:40.904902    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:41.359432 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.359396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:41.459674 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.459630    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:41.720952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.720915    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:41.752420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752390    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"]
Sep 04 22:28:41.752778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752765    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" containerName="main"
Sep 04 22:28:41.752778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752779    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" containerName="main"
Sep 04 22:28:41.752865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752791    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a43077ab-2bd6-4d17-a621-253842dcecac" containerName="maas-api"
Sep 04 22:28:41.752865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752797    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a43077ab-2bd6-4d17-a621-253842dcecac" containerName="maas-api"
Sep 04 22:28:41.752865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752813    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f6b9776-f870-421e-afc7-50a71cd347f4" containerName="main"
Sep 04 22:28:41.752865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752819    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f6b9776-f870-421e-afc7-50a71cd347f4" containerName="main"
Sep 04 22:28:41.752865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752829    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" containerName="istio-proxy"
Sep 04 22:28:41.752865 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752833    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" containerName="istio-proxy"
Sep 04 22:28:41.753049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752897    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f6b9776-f870-421e-afc7-50a71cd347f4" containerName="main"
Sep 04 22:28:41.753049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752905    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab8ecc75-0aeb-47d7-bd45-7077ab1a7f3b" containerName="main"
Sep 04 22:28:41.753049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752911    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="24bfc06b-7aec-48a8-9c9b-d5193aefd5c6" containerName="istio-proxy"
Sep 04 22:28:41.753049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.752917    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a43077ab-2bd6-4d17-a621-253842dcecac" containerName="maas-api"
Sep 04 22:28:41.757379 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.757357    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.760162 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.760142    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-bc62f0-openshift-default-dockercfg-zr59w\""
Sep 04 22:28:41.769462 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.769442    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"]
Sep 04 22:28:41.793826 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.793801    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-podinfo\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.793943 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.793840    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-socket\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.793943 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.793860    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-credential-socket\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.793943 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.793931    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8m5s\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-kube-api-access-g8m5s\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.794088 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.793958    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-certs\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.794088 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.794002    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-envoy\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.794088 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.794035    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istiod-ca-cert\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.794206 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.794128    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-token\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.794206 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.794169    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-data\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.814642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.814611    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:41.895444 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895411    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-token\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.895444 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895445    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-data\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.895867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895473    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-podinfo\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.895867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895504    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-socket\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.895867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895520    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-credential-socket\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.895867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895560    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8m5s\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-kube-api-access-g8m5s\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.895867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895703    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-certs\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.895867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895802    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-envoy\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.895867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895861    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istiod-ca-cert\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.896228 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895872    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-data\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.896228 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895922    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-socket\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.896228 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.895935    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-credential-socket\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.896344 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.896242    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-certs\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.896590 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.896569    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istiod-ca-cert\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.897924 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.897895    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-envoy\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.898139 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.898119    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-podinfo\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.904524 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.904502    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-token\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:41.906575 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:41.906555    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8m5s\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-kube-api-access-g8m5s\") pod \"e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") " pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:42.000413 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.000333    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"]
Sep 04 22:28:42.005533 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.005508    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.008225 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.008195    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a12dcf-openshift-default-dockercfg-5lnz2\""
Sep 04 22:28:42.016794 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.016761    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"]
Sep 04 22:28:42.037657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.037581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:42.068692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.068658    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:42.098708 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.098674    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-data\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.098882 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.098717    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-podinfo\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.098882 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.098810    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-certs\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.098882 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.098870    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-token\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.099076 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.098896    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0022b660-5109-4ef0-be95-c5805b6d0aef-istiod-ca-cert\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.099076 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.098934    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-credential-socket\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.099076 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.098985    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-envoy\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.099076 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.099036    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-socket\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.099247 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.099113    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68c49\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-kube-api-access-68c49\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.199776 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.199735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-token\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.199967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.199781    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0022b660-5109-4ef0-be95-c5805b6d0aef-istiod-ca-cert\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.199967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.199816    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-credential-socket\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.199967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.199853    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-envoy\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.199967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.199884    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-socket\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.199967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.199933    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68c49\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-kube-api-access-68c49\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.200274 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.199990    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-data\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.200274 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.200017    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-podinfo\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.200274 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.200099    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-certs\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.200274 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.200265    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-credential-socket\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.200477 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.200357    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-socket\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.200606 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.200576    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-data\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.200677 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.200592    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0022b660-5109-4ef0-be95-c5805b6d0aef-istiod-ca-cert\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.200677 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.200614    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-certs\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.202434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.202411    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-envoy\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.202544 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.202517    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-podinfo\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.209777 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.209747    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-token\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.210367 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.210345    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68c49\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-kube-api-access-68c49\") pod \"e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") " pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.238953 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.238922    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:42.319976 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.319885    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:42.355123 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.355097    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"]
Sep 04 22:28:42.357481 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:28:42.357453    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43cb9cfb_f543_4af3_96e4_1aa7bafe65fb.slice/crio-25f7f14049688b43470bb04373bc445d57ac33b55c793f82f7e5d571aa0d7481 WatchSource:0}: Error finding container 25f7f14049688b43470bb04373bc445d57ac33b55c793f82f7e5d571aa0d7481: Status 404 returned error can't find the container with id 25f7f14049688b43470bb04373bc445d57ac33b55c793f82f7e5d571aa0d7481
Sep 04 22:28:42.359775 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.359734    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:28:42.360100 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.359808    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:28:42.360100 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.359838    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:28:42.603182 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.603154    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"]
Sep 04 22:28:42.606406 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:28:42.606375    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0022b660_5109_4ef0_be95_c5805b6d0aef.slice/crio-35dc5770fdb14cb540f9f2e6f5857ebd46e9b4d8bc1c72e9b8d93dfc9f0ea94b WatchSource:0}: Error finding container 35dc5770fdb14cb540f9f2e6f5857ebd46e9b4d8bc1c72e9b8d93dfc9f0ea94b: Status 404 returned error can't find the container with id 35dc5770fdb14cb540f9f2e6f5857ebd46e9b4d8bc1c72e9b8d93dfc9f0ea94b
Sep 04 22:28:42.609145 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.609113    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:28:42.609234 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.609183    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:28:42.609234 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.609214    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:28:42.667336 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.667306    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:42.886312 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.886213    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" event={"ID":"0022b660-5109-4ef0-be95-c5805b6d0aef","Type":"ContainerStarted","Data":"6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1"}
Sep 04 22:28:42.886312 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.886274    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" event={"ID":"0022b660-5109-4ef0-be95-c5805b6d0aef","Type":"ContainerStarted","Data":"35dc5770fdb14cb540f9f2e6f5857ebd46e9b4d8bc1c72e9b8d93dfc9f0ea94b"}
Sep 04 22:28:42.887806 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.887779    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" event={"ID":"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb","Type":"ContainerStarted","Data":"18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd"}
Sep 04 22:28:42.887921 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.887821    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" event={"ID":"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb","Type":"ContainerStarted","Data":"25f7f14049688b43470bb04373bc445d57ac33b55c793f82f7e5d571aa0d7481"}
Sep 04 22:28:42.919254 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.918915    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" podStartSLOduration=1.918893596 podStartE2EDuration="1.918893596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:42.916458916 +0000 UTC m=+1206.435181450" watchObservedRunningTime="2026-09-04 22:28:42.918893596 +0000 UTC m=+1206.437616130"
Sep 04 22:28:42.980305 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:42.980211    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podStartSLOduration=1.9801932660000001 podStartE2EDuration="1.980193266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:42.977375217 +0000 UTC m=+1206.496097750" watchObservedRunningTime="2026-09-04 22:28:42.980193266 +0000 UTC m=+1206.498915799"
Sep 04 22:28:43.069237 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:43.069197    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:43.074486 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:43.074455    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:43.320141 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:43.320098    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:43.325411 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:43.325382    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:43.892475 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:43.892439    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:43.892928 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:43.892496    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:43.893381 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:43.893356    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:28:43.893540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:43.893514    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:28:44.932614 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:44.932582    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:45.033626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:45.033586    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:45.133223 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:45.133186    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:46.104303 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.104263    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:46.117822 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.117795    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:28:46.138793 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.138717    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-data\") pod \"dd604e6d-a4d9-4787-b133-f925d2b84367\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") "
Sep 04 22:28:46.138793 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.138750    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-credential-socket\") pod \"dd604e6d-a4d9-4787-b133-f925d2b84367\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") "
Sep 04 22:28:46.138793 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.138776    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd604e6d-a4d9-4787-b133-f925d2b84367-istiod-ca-cert\") pod \"dd604e6d-a4d9-4787-b133-f925d2b84367\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") "
Sep 04 22:28:46.139040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.138798    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-token\") pod \"dd604e6d-a4d9-4787-b133-f925d2b84367\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") "
Sep 04 22:28:46.139040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.138829    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-envoy\") pod \"dd604e6d-a4d9-4787-b133-f925d2b84367\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") "
Sep 04 22:28:46.139040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.138862    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rhtbh\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-kube-api-access-rhtbh\") pod \"dd604e6d-a4d9-4787-b133-f925d2b84367\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") "
Sep 04 22:28:46.139040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.138934    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-socket\") pod \"dd604e6d-a4d9-4787-b133-f925d2b84367\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") "
Sep 04 22:28:46.139040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.138990    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-certs\") pod \"dd604e6d-a4d9-4787-b133-f925d2b84367\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") "
Sep 04 22:28:46.139040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.139040    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-podinfo\") pod \"dd604e6d-a4d9-4787-b133-f925d2b84367\" (UID: \"dd604e6d-a4d9-4787-b133-f925d2b84367\") "
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.139082    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd604e6d-a4d9-4787-b133-f925d2b84367" (UID: "dd604e6d-a4d9-4787-b133-f925d2b84367"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.139220    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd604e6d-a4d9-4787-b133-f925d2b84367-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd604e6d-a4d9-4787-b133-f925d2b84367" (UID: "dd604e6d-a4d9-4787-b133-f925d2b84367"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.139412    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd604e6d-a4d9-4787-b133-f925d2b84367" (UID: "dd604e6d-a4d9-4787-b133-f925d2b84367"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.139487    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd604e6d-a4d9-4787-b133-f925d2b84367" (UID: "dd604e6d-a4d9-4787-b133-f925d2b84367"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.140029    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd604e6d-a4d9-4787-b133-f925d2b84367" (UID: "dd604e6d-a4d9-4787-b133-f925d2b84367"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.140222    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.140262    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-data\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.140276    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-credential-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.140289    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd604e6d-a4d9-4787-b133-f925d2b84367-istiod-ca-cert\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.140304    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-workload-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:46.142269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.141760    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd604e6d-a4d9-4787-b133-f925d2b84367" (UID: "dd604e6d-a4d9-4787-b133-f925d2b84367"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:28:46.144130 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.144093    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd604e6d-a4d9-4787-b133-f925d2b84367" (UID: "dd604e6d-a4d9-4787-b133-f925d2b84367"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:28:46.144130 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.144101    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-kube-api-access-rhtbh" (OuterVolumeSpecName: "kube-api-access-rhtbh") pod "dd604e6d-a4d9-4787-b133-f925d2b84367" (UID: "dd604e6d-a4d9-4787-b133-f925d2b84367"). InnerVolumeSpecName "kube-api-access-rhtbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:28:46.144450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.144418    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd604e6d-a4d9-4787-b133-f925d2b84367" (UID: "dd604e6d-a4d9-4787-b133-f925d2b84367"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 22:28:46.206837 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.206799    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:46.241688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.241654    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-podinfo\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:46.241688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.241678    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-token\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:46.241688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.241687    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd604e6d-a4d9-4787-b133-f925d2b84367-istio-envoy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:46.241688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.241696    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhtbh\" (UniqueName: \"kubernetes.io/projected/dd604e6d-a4d9-4787-b133-f925d2b84367-kube-api-access-rhtbh\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:28:46.343578 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.343540    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:46.714873 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.714835    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:46.815848 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.815806    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:46.917860 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.917824    2576 generic.go:356] "Generic (PLEG): container finished" podID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerID="d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7" exitCode=0
Sep 04 22:28:46.918026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.917936    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"
Sep 04 22:28:46.919451 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.919426    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" event={"ID":"dd604e6d-a4d9-4787-b133-f925d2b84367","Type":"ContainerDied","Data":"d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7"}
Sep 04 22:28:46.919542 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.919528    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s" event={"ID":"dd604e6d-a4d9-4787-b133-f925d2b84367","Type":"ContainerDied","Data":"537dc16eec6404e8006932098c413dc0c527e3ee14e2b3a71b0cf88f3361566b"}
Sep 04 22:28:46.919576 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.919565    2576 scope.go:117] "RemoveContainer" containerID="d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7"
Sep 04 22:28:46.929557 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.929543    2576 scope.go:117] "RemoveContainer" containerID="e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145"
Sep 04 22:28:46.937843 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.937826    2576 scope.go:117] "RemoveContainer" containerID="d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7"
Sep 04 22:28:46.938105 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:28:46.938082    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7\": container with ID starting with d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7 not found: ID does not exist" containerID="d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7"
Sep 04 22:28:46.938229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.938114    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7"} err="failed to get container status \"d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7\": rpc error: code = NotFound desc = could not find container \"d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7\": container with ID starting with d359e7766902f018b6587ba6717e3b3ba220df7f9f9f35b34fb4b35161343fd7 not found: ID does not exist"
Sep 04 22:28:46.938229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.938139    2576 scope.go:117] "RemoveContainer" containerID="e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145"
Sep 04 22:28:46.938403 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:28:46.938384    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145\": container with ID starting with e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145 not found: ID does not exist" containerID="e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145"
Sep 04 22:28:46.938443 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.938409    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145"} err="failed to get container status \"e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145\": rpc error: code = NotFound desc = could not find container \"e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145\": container with ID starting with e80436072219aab910d96299c1b8b9a42ffb0a53ee3066eca06e7ad3ba1ca145 not found: ID does not exist"
Sep 04 22:28:46.947289 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.947261    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"]
Sep 04 22:28:46.954264 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:46.954243    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fecd36-openshift-default-6cdcd8644b-6g28s"]
Sep 04 22:28:47.142593 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.142509    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"]
Sep 04 22:28:47.143477 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.143451    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy"
Sep 04 22:28:47.143552 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.143481    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy"
Sep 04 22:28:47.143631 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.143618    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy"
Sep 04 22:28:47.143681 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.143641    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy"
Sep 04 22:28:47.148840 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.148820    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.152163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.151793    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 22:28:47.152163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.151898    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-a12dcf-serving-cert\""
Sep 04 22:28:47.152163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.151968    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 04 22:28:47.152163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.152033    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2lmrb\""
Sep 04 22:28:47.152894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.152671    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 04 22:28:47.161202 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.161178    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"]
Sep 04 22:28:47.195967 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.195933    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:47.250254 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.250216    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71cbfce-2460-443b-82e3-9b041f946dcc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.250458 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.250285    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.250458 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.250321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.250458 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.250371    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktvxt\" (UniqueName: \"kubernetes.io/projected/f71cbfce-2460-443b-82e3-9b041f946dcc-kube-api-access-ktvxt\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.293435 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.293396    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:47.351947 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.351914    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71cbfce-2460-443b-82e3-9b041f946dcc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.352117 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.351964    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.352117 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.351985    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.352117 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.352006    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktvxt\" (UniqueName: \"kubernetes.io/projected/f71cbfce-2460-443b-82e3-9b041f946dcc-kube-api-access-ktvxt\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.352304 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:28:47.352170    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-serving-cert: secret "maas-api-e2e-shared-b-w2-a12dcf-serving-cert" not found
Sep 04 22:28:47.352304 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:28:47.352254    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-maas-api-tls podName:f71cbfce-2460-443b-82e3-9b041f946dcc nodeName:}" failed. No retries permitted until 2026-09-04 22:28:47.852233898 +0000 UTC m=+1211.370956412 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-maas-api-tls") pod "maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h" (UID: "f71cbfce-2460-443b-82e3-9b041f946dcc") : secret "maas-api-e2e-shared-b-w2-a12dcf-serving-cert" not found
Sep 04 22:28:47.352575 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.352556    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71cbfce-2460-443b-82e3-9b041f946dcc-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.354236 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.354219    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.361307 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.361285    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktvxt\" (UniqueName: \"kubernetes.io/projected/f71cbfce-2460-443b-82e3-9b041f946dcc-kube-api-access-ktvxt\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.673393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.673345    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:47.772493 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.772459    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:47.856289 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.856253    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.858680 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.858641    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:47.862616 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.862576    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:47.912688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.912665    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"]
Sep 04 22:28:47.913089 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.913075    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy"
Sep 04 22:28:47.913089 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.913089    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" containerName="istio-proxy"
Sep 04 22:28:47.916329 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.916314    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:28:47.918734 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.918708    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a12dcf\""
Sep 04 22:28:47.919041 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.919026    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a12dcf-dockercfg-5mkns\""
Sep 04 22:28:47.927219 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.927162    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"]
Sep 04 22:28:47.957503 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.957472    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fkgf\" (UniqueName: \"kubernetes.io/projected/46def525-17dd-4ecf-9f3d-5e311b85c486-kube-api-access-7fkgf\") pod \"payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl\" (UID: \"46def525-17dd-4ecf-9f3d-5e311b85c486\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:28:47.957670 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:47.957559    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46def525-17dd-4ecf-9f3d-5e311b85c486-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl\" (UID: \"46def525-17dd-4ecf-9f3d-5e311b85c486\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:28:48.058396 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.058357    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fkgf\" (UniqueName: \"kubernetes.io/projected/46def525-17dd-4ecf-9f3d-5e311b85c486-kube-api-access-7fkgf\") pod \"payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl\" (UID: \"46def525-17dd-4ecf-9f3d-5e311b85c486\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:28:48.058604 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.058454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46def525-17dd-4ecf-9f3d-5e311b85c486-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl\" (UID: \"46def525-17dd-4ecf-9f3d-5e311b85c486\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:28:48.059208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.059189    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46def525-17dd-4ecf-9f3d-5e311b85c486-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl\" (UID: \"46def525-17dd-4ecf-9f3d-5e311b85c486\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:28:48.062191 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.062170    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:48.069424 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.069401    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fkgf\" (UniqueName: \"kubernetes.io/projected/46def525-17dd-4ecf-9f3d-5e311b85c486-kube-api-access-7fkgf\") pod \"payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl\" (UID: \"46def525-17dd-4ecf-9f3d-5e311b85c486\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:28:48.166490 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.166453    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:48.228215 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.228185    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:28:48.335748 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.335687    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"]
Sep 04 22:28:48.338441 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:28:48.338416    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf71cbfce_2460_443b_82e3_9b041f946dcc.slice/crio-3575b5b574e23641e9236e57d43b5116b2dbfbdffc8e70af2e7f756d5318b28a WatchSource:0}: Error finding container 3575b5b574e23641e9236e57d43b5116b2dbfbdffc8e70af2e7f756d5318b28a: Status 404 returned error can't find the container with id 3575b5b574e23641e9236e57d43b5116b2dbfbdffc8e70af2e7f756d5318b28a
Sep 04 22:28:48.499516 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.499490    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"]
Sep 04 22:28:48.501779 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:28:48.501747    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46def525_17dd_4ecf_9f3d_5e311b85c486.slice/crio-07450ffc6d65b11f55d172049f2fba4530efcb559548d7ae9e55340abca8de39 WatchSource:0}: Error finding container 07450ffc6d65b11f55d172049f2fba4530efcb559548d7ae9e55340abca8de39: Status 404 returned error can't find the container with id 07450ffc6d65b11f55d172049f2fba4530efcb559548d7ae9e55340abca8de39
Sep 04 22:28:48.920218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.920136    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd604e6d-a4d9-4787-b133-f925d2b84367" path="/var/lib/kubelet/pods/dd604e6d-a4d9-4787-b133-f925d2b84367/volumes"
Sep 04 22:28:48.931325 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.931290    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl" event={"ID":"46def525-17dd-4ecf-9f3d-5e311b85c486","Type":"ContainerStarted","Data":"005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27"}
Sep 04 22:28:48.931501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.931343    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl" event={"ID":"46def525-17dd-4ecf-9f3d-5e311b85c486","Type":"ContainerStarted","Data":"07450ffc6d65b11f55d172049f2fba4530efcb559548d7ae9e55340abca8de39"}
Sep 04 22:28:48.931501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.931372    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:28:48.932837 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.932813    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h" event={"ID":"f71cbfce-2460-443b-82e3-9b041f946dcc","Type":"ContainerStarted","Data":"9568f32b34949808fcc5718bc029d7af84ff5b153916bf6f082a0f6a6edc19b6"}
Sep 04 22:28:48.932952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.932861    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h" event={"ID":"f71cbfce-2460-443b-82e3-9b041f946dcc","Type":"ContainerStarted","Data":"3575b5b574e23641e9236e57d43b5116b2dbfbdffc8e70af2e7f756d5318b28a"}
Sep 04 22:28:48.932952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.932912    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:48.948907 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.948856    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl" podStartSLOduration=1.94883986 podStartE2EDuration="1.94883986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:48.947923831 +0000 UTC m=+1212.466646364" watchObservedRunningTime="2026-09-04 22:28:48.94883986 +0000 UTC m=+1212.467562433"
Sep 04 22:28:48.972081 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:48.967866    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h" podStartSLOduration=1.682839787 podStartE2EDuration="1.967844781s" totalImagesPullingTime="285.004994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:28:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:28:48.964276296 +0000 UTC m=+1212.482998830" watchObservedRunningTime="2026-09-04 22:28:48.967844781 +0000 UTC m=+1212.486567315"
Sep 04 22:28:49.199096 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:49.198980    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:49.901400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:49.901363    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:28:53.991886 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:53.991838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:28:54.945461 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:54.945432    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:28:59.893595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:59.892820    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:40396->10.133.0.53:15021: read: connection reset by peer (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 22:28:59.893595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:59.892891    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": read tcp 10.133.0.2:40396->10.133.0.53:15021: read: connection reset by peer (Client.Timeout exceeded while awaiting headers)"
Sep 04 22:28:59.934202 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:59.934169    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 04 22:28:59.934321 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:59.934237    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 04 22:28:59.941790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:28:59.941768    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:29:00.007294 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:00.007260    2576 generic.go:356] "Generic (PLEG): container finished" podID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerID="18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd" exitCode=0
Sep 04 22:29:00.007294 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:00.007292    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" event={"ID":"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb","Type":"ContainerDied","Data":"18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd"}
Sep 04 22:29:00.007717 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:00.007701    2576 scope.go:117] "RemoveContainer" containerID="18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd"
Sep 04 22:29:00.009063 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:00.009012    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:00.009182 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:00.009135    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:00.009246 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:00.009186    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:01.014456 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.014424    2576 generic.go:356] "Generic (PLEG): container finished" podID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerID="6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1" exitCode=0
Sep 04 22:29:01.014933 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.014498    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" event={"ID":"0022b660-5109-4ef0-be95-c5805b6d0aef","Type":"ContainerDied","Data":"6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1"}
Sep 04 22:29:01.015085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.015066    2576 scope.go:117] "RemoveContainer" containerID="6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1"
Sep 04 22:29:01.016254 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.016212    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:01.016363 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.016287    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:01.016363 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.016333    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:01.016724 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.016705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" event={"ID":"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb","Type":"ContainerStarted","Data":"139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be"}
Sep 04 22:29:01.029683 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.029651    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:01.069398 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.069354    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:29:01.069595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.069575    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 04 22:29:01.069671 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:01.069631    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 04 22:29:02.024502 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:02.024461    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" event={"ID":"0022b660-5109-4ef0-be95-c5805b6d0aef","Type":"ContainerStarted","Data":"6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5"}
Sep 04 22:29:02.069405 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:02.069358    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 04 22:29:02.069614 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:02.069437    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 04 22:29:02.320468 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:02.320371    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:29:02.320643 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:02.320619    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 04 22:29:02.320691 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:02.320671    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 04 22:29:03.069952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:03.069899    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 04 22:29:03.070482 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:03.069977    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 04 22:29:03.320976 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:03.320887    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 04 22:29:03.320976 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:03.320945    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 04 22:29:04.069544 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:04.069501    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 04 22:29:04.069899 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:04.069868    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 04 22:29:04.320514 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:04.320417    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 04 22:29:04.320514 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:04.320476    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 04 22:29:05.069761 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:05.069713    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 04 22:29:05.070123 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:05.069789    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 04 22:29:05.321222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:05.321127    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 04 22:29:05.321222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:05.321191    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 04 22:29:06.070003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:06.069955    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused" start-of-body=
Sep 04 22:29:06.070232 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:06.070027    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.53:15021/healthz/ready\": dial tcp 10.133.0.53:15021: connect: connection refused"
Sep 04 22:29:06.321201 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:06.321117    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused" start-of-body=
Sep 04 22:29:06.321201 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:06.321173    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.54:15021/healthz/ready\": dial tcp 10.133.0.54:15021: connect: connection refused"
Sep 04 22:29:07.072542 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:07.072508    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:29:07.072829 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:07.072801    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:29:07.073590 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:07.073559    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:29:07.160729 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:07.160616    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:07.452587 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:07.452489    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:29:07.453085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:07.452836    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:29:07.453750 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:07.453726    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:29:08.736687 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:08.736655    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:09.543944 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:09.543890    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:09.997849 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:09.997817    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:10.098917 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:10.098882    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:10.374302 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:10.374205    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:10.495137 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:10.495094    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:10.714131 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:10.714100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:10.900762 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:10.900727    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7740-369d3a"]
Sep 04 22:29:10.904288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:10.904270    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-369d3a"
Sep 04 22:29:10.908350 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:10.908243    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dlrbm\""
Sep 04 22:29:10.921370 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:10.921347    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7740-369d3a"]
Sep 04 22:29:11.000804 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:11.000697    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp4kg\" (UniqueName: \"kubernetes.io/projected/472ffe24-0e9a-4582-8b04-f8fcd08fb55b-kube-api-access-bp4kg\") pod \"test-curl-7740-369d3a\" (UID: \"472ffe24-0e9a-4582-8b04-f8fcd08fb55b\") " pod="opendatahub/test-curl-7740-369d3a"
Sep 04 22:29:11.102018 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:11.101971    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp4kg\" (UniqueName: \"kubernetes.io/projected/472ffe24-0e9a-4582-8b04-f8fcd08fb55b-kube-api-access-bp4kg\") pod \"test-curl-7740-369d3a\" (UID: \"472ffe24-0e9a-4582-8b04-f8fcd08fb55b\") " pod="opendatahub/test-curl-7740-369d3a"
Sep 04 22:29:11.112862 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:11.112826    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp4kg\" (UniqueName: \"kubernetes.io/projected/472ffe24-0e9a-4582-8b04-f8fcd08fb55b-kube-api-access-bp4kg\") pod \"test-curl-7740-369d3a\" (UID: \"472ffe24-0e9a-4582-8b04-f8fcd08fb55b\") " pod="opendatahub/test-curl-7740-369d3a"
Sep 04 22:29:11.216320 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:11.216282    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-369d3a"
Sep 04 22:29:11.417212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:11.417175    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:11.486473 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:11.486447    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7740-369d3a"]
Sep 04 22:29:11.488382 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:29:11.488356    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod472ffe24_0e9a_4582_8b04_f8fcd08fb55b.slice/crio-63bc82b4aad5044b70a26869d5f4b8710d59e812df4d0372466f0c1737776e20 WatchSource:0}: Error finding container 63bc82b4aad5044b70a26869d5f4b8710d59e812df4d0372466f0c1737776e20: Status 404 returned error can't find the container with id 63bc82b4aad5044b70a26869d5f4b8710d59e812df4d0372466f0c1737776e20
Sep 04 22:29:12.086024 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:12.085984    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7740-369d3a" event={"ID":"472ffe24-0e9a-4582-8b04-f8fcd08fb55b","Type":"ContainerStarted","Data":"e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f"}
Sep 04 22:29:12.086445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:12.086036    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7740-369d3a" event={"ID":"472ffe24-0e9a-4582-8b04-f8fcd08fb55b","Type":"ContainerStarted","Data":"63bc82b4aad5044b70a26869d5f4b8710d59e812df4d0372466f0c1737776e20"}
Sep 04 22:29:12.103336 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:12.103281    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7740-369d3a" podStartSLOduration=2.103264225 podStartE2EDuration="2.103264225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:12.10170988 +0000 UTC m=+1235.620432414" watchObservedRunningTime="2026-09-04 22:29:12.103264225 +0000 UTC m=+1235.621986757"
Sep 04 22:29:12.802025 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:12.801989    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7740-369d3a"]
Sep 04 22:29:12.806792 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:12.806717    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7740-369d3a"]
Sep 04 22:29:13.052221 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.052180    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7740-c85c9d"]
Sep 04 22:29:13.052641 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.052622    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="472ffe24-0e9a-4582-8b04-f8fcd08fb55b" containerName="test-curl-7740-369d3a"
Sep 04 22:29:13.052697 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.052643    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="472ffe24-0e9a-4582-8b04-f8fcd08fb55b" containerName="test-curl-7740-369d3a"
Sep 04 22:29:13.052738 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.052707    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="472ffe24-0e9a-4582-8b04-f8fcd08fb55b" containerName="test-curl-7740-369d3a"
Sep 04 22:29:13.055931 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.055914    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-c85c9d"
Sep 04 22:29:13.064041 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.063721    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7740-c85c9d"]
Sep 04 22:29:13.090973 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:29:13.090935    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7740-369d3a" container="test-curl-7740-369d3a"
Sep 04 22:29:13.222482 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.222442    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k85d\" (UniqueName: \"kubernetes.io/projected/20fa2c93-8c9e-4264-837d-47be6d4362ac-kube-api-access-8k85d\") pod \"test-curl-7740-c85c9d\" (UID: \"20fa2c93-8c9e-4264-837d-47be6d4362ac\") " pod="opendatahub/test-curl-7740-c85c9d"
Sep 04 22:29:13.324038 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.323931    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k85d\" (UniqueName: \"kubernetes.io/projected/20fa2c93-8c9e-4264-837d-47be6d4362ac-kube-api-access-8k85d\") pod \"test-curl-7740-c85c9d\" (UID: \"20fa2c93-8c9e-4264-837d-47be6d4362ac\") " pod="opendatahub/test-curl-7740-c85c9d"
Sep 04 22:29:13.333351 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.333316    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k85d\" (UniqueName: \"kubernetes.io/projected/20fa2c93-8c9e-4264-837d-47be6d4362ac-kube-api-access-8k85d\") pod \"test-curl-7740-c85c9d\" (UID: \"20fa2c93-8c9e-4264-837d-47be6d4362ac\") " pod="opendatahub/test-curl-7740-c85c9d"
Sep 04 22:29:13.367320 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.367278    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-c85c9d"
Sep 04 22:29:13.638597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.638570    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7740-c85c9d"]
Sep 04 22:29:13.640703 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:29:13.640675    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod20fa2c93_8c9e_4264_837d_47be6d4362ac.slice/crio-8bab0544b51003b359134dae27bfc997fc6030c18ab0f98d6940c54ab1d34fe1 WatchSource:0}: Error finding container 8bab0544b51003b359134dae27bfc997fc6030c18ab0f98d6940c54ab1d34fe1: Status 404 returned error can't find the container with id 8bab0544b51003b359134dae27bfc997fc6030c18ab0f98d6940c54ab1d34fe1
Sep 04 22:29:13.759974 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.759941    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:13.859410 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:13.859380    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:14.097199 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.097161    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7740-c85c9d" event={"ID":"20fa2c93-8c9e-4264-837d-47be6d4362ac","Type":"ContainerStarted","Data":"6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2"}
Sep 04 22:29:14.097642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.097209    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7740-c85c9d" event={"ID":"20fa2c93-8c9e-4264-837d-47be6d4362ac","Type":"ContainerStarted","Data":"8bab0544b51003b359134dae27bfc997fc6030c18ab0f98d6940c54ab1d34fe1"}
Sep 04 22:29:14.097642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.097286    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7740-369d3a" podUID="472ffe24-0e9a-4582-8b04-f8fcd08fb55b" containerName="test-curl-7740-369d3a" containerID="cri-o://e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f" gracePeriod=2
Sep 04 22:29:14.116298 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.116246    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7740-c85c9d" podStartSLOduration=1.116228559 podStartE2EDuration="1.116228559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:14.114606982 +0000 UTC m=+1237.633329516" watchObservedRunningTime="2026-09-04 22:29:14.116228559 +0000 UTC m=+1237.634951093"
Sep 04 22:29:14.751254 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.751220    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7740-c85c9d"]
Sep 04 22:29:14.756207 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.756171    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7740-c85c9d"]
Sep 04 22:29:14.894109 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.893999    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"]
Sep 04 22:29:14.894700 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.894683    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20fa2c93-8c9e-4264-837d-47be6d4362ac" containerName="test-curl-7740-c85c9d"
Sep 04 22:29:14.894749 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.894703    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="20fa2c93-8c9e-4264-837d-47be6d4362ac" containerName="test-curl-7740-c85c9d"
Sep 04 22:29:14.894803 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.894791    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="20fa2c93-8c9e-4264-837d-47be6d4362ac" containerName="test-curl-7740-c85c9d"
Sep 04 22:29:14.904285 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.904252    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:14.907072 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.906891    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-bf86db-serving-cert\""
Sep 04 22:29:14.908579 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.908550    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"]
Sep 04 22:29:14.933737 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.933695    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:14.990732 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.990700    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7740-cbb116"]
Sep 04 22:29:14.994271 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:14.994254    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-cbb116"
Sep 04 22:29:15.004010 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.002699    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7740-cbb116"]
Sep 04 22:29:15.006180 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.005827    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:15.043922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.043883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgvls\" (UniqueName: \"kubernetes.io/projected/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-kube-api-access-sgvls\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.043922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.043925    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.044168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.044009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.044168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.044095    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.088124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.088079    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:15.102708 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:29:15.102679    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7740-c85c9d" container="test-curl-7740-c85c9d"
Sep 04 22:29:15.145440 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.145343    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.145440 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.145389    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.145653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.145447    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmc97\" (UniqueName: \"kubernetes.io/projected/c01f954a-354a-42a1-a85d-655fa19b0c82-kube-api-access-cmc97\") pod \"test-curl-7740-cbb116\" (UID: \"c01f954a-354a-42a1-a85d-655fa19b0c82\") " pod="opendatahub/test-curl-7740-cbb116"
Sep 04 22:29:15.145653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.145505    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgvls\" (UniqueName: \"kubernetes.io/projected/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-kube-api-access-sgvls\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.145653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.145532    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.146120 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.146097    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.147989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.147966    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.148096 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.148013    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.154683 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.154663    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgvls\" (UniqueName: \"kubernetes.io/projected/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-kube-api-access-sgvls\") pod \"maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.217638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.217594    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:15.246212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.246163    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmc97\" (UniqueName: \"kubernetes.io/projected/c01f954a-354a-42a1-a85d-655fa19b0c82-kube-api-access-cmc97\") pod \"test-curl-7740-cbb116\" (UID: \"c01f954a-354a-42a1-a85d-655fa19b0c82\") " pod="opendatahub/test-curl-7740-cbb116"
Sep 04 22:29:15.256479 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.256450    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmc97\" (UniqueName: \"kubernetes.io/projected/c01f954a-354a-42a1-a85d-655fa19b0c82-kube-api-access-cmc97\") pod \"test-curl-7740-cbb116\" (UID: \"c01f954a-354a-42a1-a85d-655fa19b0c82\") " pod="opendatahub/test-curl-7740-cbb116"
Sep 04 22:29:15.310280 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.310244    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-cbb116"
Sep 04 22:29:15.435366 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.435283    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:15.492124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.492100    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"]
Sep 04 22:29:15.494230 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:29:15.494203    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1b00a0_b8dd_4db4_a93d_7bb0c160560e.slice/crio-27a71fbc92fbba6fc73a68fae9979d337bfcfd6d59e346b526aa8cc1b3b02e14 WatchSource:0}: Error finding container 27a71fbc92fbba6fc73a68fae9979d337bfcfd6d59e346b526aa8cc1b3b02e14: Status 404 returned error can't find the container with id 27a71fbc92fbba6fc73a68fae9979d337bfcfd6d59e346b526aa8cc1b3b02e14
Sep 04 22:29:15.535100 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.535038    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:15.598445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:15.598404    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7740-cbb116"]
Sep 04 22:29:15.600225 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:29:15.600191    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc01f954a_354a_42a1_a85d_655fa19b0c82.slice/crio-54366aa4bc6998819064898c90f28cf9ce04b7ee35a1f31a35eb4b26242ee1a7 WatchSource:0}: Error finding container 54366aa4bc6998819064898c90f28cf9ce04b7ee35a1f31a35eb4b26242ee1a7: Status 404 returned error can't find the container with id 54366aa4bc6998819064898c90f28cf9ce04b7ee35a1f31a35eb4b26242ee1a7
Sep 04 22:29:16.109269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.109234    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7740-cbb116" event={"ID":"c01f954a-354a-42a1-a85d-655fa19b0c82","Type":"ContainerStarted","Data":"fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b"}
Sep 04 22:29:16.109570 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.109289    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7740-cbb116" event={"ID":"c01f954a-354a-42a1-a85d-655fa19b0c82","Type":"ContainerStarted","Data":"54366aa4bc6998819064898c90f28cf9ce04b7ee35a1f31a35eb4b26242ee1a7"}
Sep 04 22:29:16.110492 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.110470    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4" event={"ID":"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e","Type":"ContainerStarted","Data":"27a71fbc92fbba6fc73a68fae9979d337bfcfd6d59e346b526aa8cc1b3b02e14"}
Sep 04 22:29:16.110575 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.110523    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7740-c85c9d" podUID="20fa2c93-8c9e-4264-837d-47be6d4362ac" containerName="test-curl-7740-c85c9d" containerID="cri-o://6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2" gracePeriod=2
Sep 04 22:29:16.123600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.123549    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7740-cbb116" podStartSLOduration=2.123532752 podStartE2EDuration="2.123532752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:16.12321241 +0000 UTC m=+1239.641934944" watchObservedRunningTime="2026-09-04 22:29:16.123532752 +0000 UTC m=+1239.642255284"
Sep 04 22:29:16.274030 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.273998    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-369d3a"
Sep 04 22:29:16.458836 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.458805    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bp4kg\" (UniqueName: \"kubernetes.io/projected/472ffe24-0e9a-4582-8b04-f8fcd08fb55b-kube-api-access-bp4kg\") pod \"472ffe24-0e9a-4582-8b04-f8fcd08fb55b\" (UID: \"472ffe24-0e9a-4582-8b04-f8fcd08fb55b\") "
Sep 04 22:29:16.460869 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.460844    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/472ffe24-0e9a-4582-8b04-f8fcd08fb55b-kube-api-access-bp4kg" (OuterVolumeSpecName: "kube-api-access-bp4kg") pod "472ffe24-0e9a-4582-8b04-f8fcd08fb55b" (UID: "472ffe24-0e9a-4582-8b04-f8fcd08fb55b"). InnerVolumeSpecName "kube-api-access-bp4kg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:29:16.560453 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.560427    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bp4kg\" (UniqueName: \"kubernetes.io/projected/472ffe24-0e9a-4582-8b04-f8fcd08fb55b-kube-api-access-bp4kg\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:29:16.746417 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.746328    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7740-cbb116"]
Sep 04 22:29:16.751086 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.751043    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7740-cbb116"]
Sep 04 22:29:16.918836 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.918795    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="472ffe24-0e9a-4582-8b04-f8fcd08fb55b" path="/var/lib/kubelet/pods/472ffe24-0e9a-4582-8b04-f8fcd08fb55b/volumes"
Sep 04 22:29:16.998020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.997948    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7740-115688"]
Sep 04 22:29:16.998408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.998395    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c01f954a-354a-42a1-a85d-655fa19b0c82" containerName="test-curl-7740-cbb116"
Sep 04 22:29:16.998458 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.998410    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="c01f954a-354a-42a1-a85d-655fa19b0c82" containerName="test-curl-7740-cbb116"
Sep 04 22:29:16.998495 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:16.998470    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="c01f954a-354a-42a1-a85d-655fa19b0c82" containerName="test-curl-7740-cbb116"
Sep 04 22:29:17.001672 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.001655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-115688"
Sep 04 22:29:17.006468 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.006444    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7740-115688"]
Sep 04 22:29:17.064971 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.064937    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xmbb\" (UniqueName: \"kubernetes.io/projected/8a7e1002-9fa7-417e-a593-c8ffdb497fb4-kube-api-access-2xmbb\") pod \"test-curl-7740-115688\" (UID: \"8a7e1002-9fa7-417e-a593-c8ffdb497fb4\") " pod="opendatahub/test-curl-7740-115688"
Sep 04 22:29:17.117163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.117127    2576 generic.go:356] "Generic (PLEG): container finished" podID="472ffe24-0e9a-4582-8b04-f8fcd08fb55b" containerID="e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f" exitCode=137
Sep 04 22:29:17.117658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.117175    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-369d3a"
Sep 04 22:29:17.117658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.117231    2576 scope.go:117] "RemoveContainer" containerID="e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f"
Sep 04 22:29:17.119120 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.119091    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4" event={"ID":"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e","Type":"ContainerStarted","Data":"478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8"}
Sep 04 22:29:17.119230 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:29:17.119215    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7740-cbb116" container="test-curl-7740-cbb116"
Sep 04 22:29:17.126116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.126099    2576 scope.go:117] "RemoveContainer" containerID="e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f"
Sep 04 22:29:17.126387 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:29:17.126370    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f\": container with ID starting with e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f not found: ID does not exist" containerID="e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f"
Sep 04 22:29:17.126460 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.126396    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f"} err="failed to get container status \"e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f\": rpc error: code = NotFound desc = could not find container \"e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f\": container with ID starting with e2a052e5c85d14028bec50230339d61e9d678abda1e7230bcc6ba227cf07e59f not found: ID does not exist"
Sep 04 22:29:17.166386 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.166342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xmbb\" (UniqueName: \"kubernetes.io/projected/8a7e1002-9fa7-417e-a593-c8ffdb497fb4-kube-api-access-2xmbb\") pod \"test-curl-7740-115688\" (UID: \"8a7e1002-9fa7-417e-a593-c8ffdb497fb4\") " pod="opendatahub/test-curl-7740-115688"
Sep 04 22:29:17.180513 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.180480    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xmbb\" (UniqueName: \"kubernetes.io/projected/8a7e1002-9fa7-417e-a593-c8ffdb497fb4-kube-api-access-2xmbb\") pod \"test-curl-7740-115688\" (UID: \"8a7e1002-9fa7-417e-a593-c8ffdb497fb4\") " pod="opendatahub/test-curl-7740-115688"
Sep 04 22:29:17.314049 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.313966    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-115688"
Sep 04 22:29:17.586747 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.586634    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4" podStartSLOduration=3.016333009 podStartE2EDuration="3.58661309s" totalImagesPullingTime="570.280081ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:17.1578949 +0000 UTC m=+1240.676617433" watchObservedRunningTime="2026-09-04 22:29:17.58661309 +0000 UTC m=+1241.105335624"
Sep 04 22:29:17.589151 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:17.589128    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7740-115688"]
Sep 04 22:29:17.591127 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:29:17.591101    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a7e1002_9fa7_417e_a593_c8ffdb497fb4.slice/crio-5f425468fa4b9b6d3a762574e654d713b4c5026b8b18281b700b0a945e08df03 WatchSource:0}: Error finding container 5f425468fa4b9b6d3a762574e654d713b4c5026b8b18281b700b0a945e08df03: Status 404 returned error can't find the container with id 5f425468fa4b9b6d3a762574e654d713b4c5026b8b18281b700b0a945e08df03
Sep 04 22:29:18.126292 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.126260    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7740-115688" event={"ID":"8a7e1002-9fa7-417e-a593-c8ffdb497fb4","Type":"ContainerStarted","Data":"3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de"}
Sep 04 22:29:18.126675 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.126307    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7740-115688" event={"ID":"8a7e1002-9fa7-417e-a593-c8ffdb497fb4","Type":"ContainerStarted","Data":"5f425468fa4b9b6d3a762574e654d713b4c5026b8b18281b700b0a945e08df03"}
Sep 04 22:29:18.126675 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.126543    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:18.126675 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.126611    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7740-cbb116" podUID="c01f954a-354a-42a1-a85d-655fa19b0c82" containerName="test-curl-7740-cbb116" containerID="cri-o://fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b" gracePeriod=2
Sep 04 22:29:18.142999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.142957    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7740-115688" podStartSLOduration=2.14294272 podStartE2EDuration="2.14294272s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:18.141496048 +0000 UTC m=+1241.660218582" watchObservedRunningTime="2026-09-04 22:29:18.14294272 +0000 UTC m=+1241.661665252"
Sep 04 22:29:18.301259 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.301234    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-c85c9d"
Sep 04 22:29:18.376194 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.376110    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k85d\" (UniqueName: \"kubernetes.io/projected/20fa2c93-8c9e-4264-837d-47be6d4362ac-kube-api-access-8k85d\") pod \"20fa2c93-8c9e-4264-837d-47be6d4362ac\" (UID: \"20fa2c93-8c9e-4264-837d-47be6d4362ac\") "
Sep 04 22:29:18.378169 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.378142    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fa2c93-8c9e-4264-837d-47be6d4362ac-kube-api-access-8k85d" (OuterVolumeSpecName: "kube-api-access-8k85d") pod "20fa2c93-8c9e-4264-837d-47be6d4362ac" (UID: "20fa2c93-8c9e-4264-837d-47be6d4362ac"). InnerVolumeSpecName "kube-api-access-8k85d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:29:18.477429 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.477393    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k85d\" (UniqueName: \"kubernetes.io/projected/20fa2c93-8c9e-4264-837d-47be6d4362ac-kube-api-access-8k85d\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:29:18.756245 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.756212    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7740-115688"]
Sep 04 22:29:18.767806 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.767752    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7740-115688"]
Sep 04 22:29:18.918682 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:18.918646    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20fa2c93-8c9e-4264-837d-47be6d4362ac" path="/var/lib/kubelet/pods/20fa2c93-8c9e-4264-837d-47be6d4362ac/volumes"
Sep 04 22:29:19.132653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.132555    2576 generic.go:356] "Generic (PLEG): container finished" podID="20fa2c93-8c9e-4264-837d-47be6d4362ac" containerID="6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2" exitCode=137
Sep 04 22:29:19.132653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.132617    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-c85c9d"
Sep 04 22:29:19.133175 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.132661    2576 scope.go:117] "RemoveContainer" containerID="6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2"
Sep 04 22:29:19.133175 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:29:19.133096    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7740-115688" container="test-curl-7740-115688"
Sep 04 22:29:19.141452 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.141434    2576 scope.go:117] "RemoveContainer" containerID="6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2"
Sep 04 22:29:19.141722 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:29:19.141703    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2\": container with ID starting with 6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2 not found: ID does not exist" containerID="6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2"
Sep 04 22:29:19.141771 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.141729    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2"} err="failed to get container status \"6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2\": rpc error: code = NotFound desc = could not find container \"6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2\": container with ID starting with 6642ba1114abd75f9c231685d426e9eb2f5226806985cbdaf8a7f0aca746f3b2 not found: ID does not exist"
Sep 04 22:29:19.430899 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.430804    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:19.704519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.704492    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"]
Sep 04 22:29:19.704962 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.704948    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a7e1002-9fa7-417e-a593-c8ffdb497fb4" containerName="test-curl-7740-115688"
Sep 04 22:29:19.705007 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.704965    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a7e1002-9fa7-417e-a593-c8ffdb497fb4" containerName="test-curl-7740-115688"
Sep 04 22:29:19.705074 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.705049    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a7e1002-9fa7-417e-a593-c8ffdb497fb4" containerName="test-curl-7740-115688"
Sep 04 22:29:19.709775 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.709753    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.712404 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.712380    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-3949cd16-openshift-default-dockercfg-9s9j8\""
Sep 04 22:29:19.719966 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.719940    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"]
Sep 04 22:29:19.738586 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.738550    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:19.787764 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.787721    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-credential-socket\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.787927 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.787778    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-certs\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.787927 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.787817    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-envoy\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.787927 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.787851    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istiod-ca-cert\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.787927 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.787878    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrbbc\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-kube-api-access-lrbbc\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.787927 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.787910    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-data\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.788175 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.787952    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-socket\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.788175 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.787987    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-token\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.788175 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.788089    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-podinfo\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.888632 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.888588    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-data\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.888814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.888653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-socket\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.888814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.888688    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-token\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.888814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.888748    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-podinfo\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.888814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.888805    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-credential-socket\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.889040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.888855    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-certs\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.889040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.888892    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-envoy\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.889040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.888938    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istiod-ca-cert\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.889040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.888969    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrbbc\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-kube-api-access-lrbbc\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.889288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.889043    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-data\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.889288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.889146    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-socket\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.889288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.889237    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-certs\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.889538 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.889513    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-credential-socket\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.889754 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.889735    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istiod-ca-cert\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.891128 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.891110    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-envoy\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.891352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.891334    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-podinfo\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.897570 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.897546    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-token\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:19.897885 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:19.897863    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrbbc\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-kube-api-access-lrbbc\") pod \"e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") " pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:20.002550 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.002472    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:20.022402 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.022361    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:20.139778 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.139740    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7740-115688" podUID="8a7e1002-9fa7-417e-a593-c8ffdb497fb4" containerName="test-curl-7740-115688" containerID="cri-o://3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de" gracePeriod=2
Sep 04 22:29:20.296997 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.296973    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-cbb116"
Sep 04 22:29:20.301438 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.301411    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"]
Sep 04 22:29:20.303103 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:29:20.303073    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97d3dcd1_941b_44bc_a405_9b1c774e84d4.slice/crio-3bf260f93eb2fc9ac5611f377933abc45825886c0328c2708a061b8f015709c8 WatchSource:0}: Error finding container 3bf260f93eb2fc9ac5611f377933abc45825886c0328c2708a061b8f015709c8: Status 404 returned error can't find the container with id 3bf260f93eb2fc9ac5611f377933abc45825886c0328c2708a061b8f015709c8
Sep 04 22:29:20.305393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.305358    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:20.305483 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.305426    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:20.305483 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.305457    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:20.392422 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.392392    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cmc97\" (UniqueName: \"kubernetes.io/projected/c01f954a-354a-42a1-a85d-655fa19b0c82-kube-api-access-cmc97\") pod \"c01f954a-354a-42a1-a85d-655fa19b0c82\" (UID: \"c01f954a-354a-42a1-a85d-655fa19b0c82\") "
Sep 04 22:29:20.394329 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.394304    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c01f954a-354a-42a1-a85d-655fa19b0c82-kube-api-access-cmc97" (OuterVolumeSpecName: "kube-api-access-cmc97") pod "c01f954a-354a-42a1-a85d-655fa19b0c82" (UID: "c01f954a-354a-42a1-a85d-655fa19b0c82"). InnerVolumeSpecName "kube-api-access-cmc97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:29:20.493162 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.493129    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmc97\" (UniqueName: \"kubernetes.io/projected/c01f954a-354a-42a1-a85d-655fa19b0c82-kube-api-access-cmc97\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:29:20.918367 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:20.918334    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c01f954a-354a-42a1-a85d-655fa19b0c82" path="/var/lib/kubelet/pods/c01f954a-354a-42a1-a85d-655fa19b0c82/volumes"
Sep 04 22:29:21.146610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:21.146572    2576 generic.go:356] "Generic (PLEG): container finished" podID="c01f954a-354a-42a1-a85d-655fa19b0c82" containerID="fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b" exitCode=137
Sep 04 22:29:21.147097 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:21.146616    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-cbb116"
Sep 04 22:29:21.147097 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:21.146676    2576 scope.go:117] "RemoveContainer" containerID="fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b"
Sep 04 22:29:21.148846 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:21.148781    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" event={"ID":"97d3dcd1-941b-44bc-a405-9b1c774e84d4","Type":"ContainerStarted","Data":"0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e"}
Sep 04 22:29:21.148956 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:21.148882    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" event={"ID":"97d3dcd1-941b-44bc-a405-9b1c774e84d4","Type":"ContainerStarted","Data":"3bf260f93eb2fc9ac5611f377933abc45825886c0328c2708a061b8f015709c8"}
Sep 04 22:29:21.155519 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:21.155504    2576 scope.go:117] "RemoveContainer" containerID="fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b"
Sep 04 22:29:21.155771 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:29:21.155755    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b\": container with ID starting with fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b not found: ID does not exist" containerID="fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b"
Sep 04 22:29:21.155814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:21.155777    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b"} err="failed to get container status \"fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b\": rpc error: code = NotFound desc = could not find container \"fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b\": container with ID starting with fa7b30f366525989e67d994ae534e8423a03809fa9cdaadd20e20d2b100a472b not found: ID does not exist"
Sep 04 22:29:21.171388 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:21.171350    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" podStartSLOduration=2.171335796 podStartE2EDuration="2.171335796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:21.170724288 +0000 UTC m=+1244.689446821" watchObservedRunningTime="2026-09-04 22:29:21.171335796 +0000 UTC m=+1244.690058329"
Sep 04 22:29:22.022688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.022646    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:22.028119 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.028092    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:22.161392 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.161362    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:22.162298 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.162280    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:22.344609 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.344582    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-115688"
Sep 04 22:29:22.411079 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.411030    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xmbb\" (UniqueName: \"kubernetes.io/projected/8a7e1002-9fa7-417e-a593-c8ffdb497fb4-kube-api-access-2xmbb\") pod \"8a7e1002-9fa7-417e-a593-c8ffdb497fb4\" (UID: \"8a7e1002-9fa7-417e-a593-c8ffdb497fb4\") "
Sep 04 22:29:22.412969 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.412944    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a7e1002-9fa7-417e-a593-c8ffdb497fb4-kube-api-access-2xmbb" (OuterVolumeSpecName: "kube-api-access-2xmbb") pod "8a7e1002-9fa7-417e-a593-c8ffdb497fb4" (UID: "8a7e1002-9fa7-417e-a593-c8ffdb497fb4"). InnerVolumeSpecName "kube-api-access-2xmbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:29:22.511952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.511908    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xmbb\" (UniqueName: \"kubernetes.io/projected/8a7e1002-9fa7-417e-a593-c8ffdb497fb4-kube-api-access-2xmbb\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:29:22.602008 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.601926    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:29:22.919116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:22.919020    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a7e1002-9fa7-417e-a593-c8ffdb497fb4" path="/var/lib/kubelet/pods/8a7e1002-9fa7-417e-a593-c8ffdb497fb4/volumes"
Sep 04 22:29:23.168177 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:23.168136    2576 generic.go:356] "Generic (PLEG): container finished" podID="8a7e1002-9fa7-417e-a593-c8ffdb497fb4" containerID="3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de" exitCode=137
Sep 04 22:29:23.168672 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:23.168188    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7740-115688"
Sep 04 22:29:23.168672 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:23.168252    2576 scope.go:117] "RemoveContainer" containerID="3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de"
Sep 04 22:29:23.177690 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:23.177671    2576 scope.go:117] "RemoveContainer" containerID="3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de"
Sep 04 22:29:23.177975 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:29:23.177953    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de\": container with ID starting with 3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de not found: ID does not exist" containerID="3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de"
Sep 04 22:29:23.178076 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:23.177983    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de"} err="failed to get container status \"3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de\": rpc error: code = NotFound desc = could not find container \"3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de\": container with ID starting with 3568ee5f878930ed35578360f409bfbe8f773be73194d6dbf5ae14ceb06038de not found: ID does not exist"
Sep 04 22:29:23.218337 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:23.218294    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:23.309236 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:23.309190    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:24.138882 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:24.138850    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:29:24.499517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:24.499478    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:24.599273 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:24.599242    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:24.864644 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:24.864545    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:25.022006 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:25.021974    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:25.125163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:25.125041    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:35.983031 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:35.982997    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:38.973422 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:38.973389    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:40.401069 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:40.401008    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:29:43.278607 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:43.278570    2576 generic.go:356] "Generic (PLEG): container finished" podID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerID="0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e" exitCode=0
Sep 04 22:29:43.279106 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:43.278652    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" event={"ID":"97d3dcd1-941b-44bc-a405-9b1c774e84d4","Type":"ContainerDied","Data":"0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e"}
Sep 04 22:29:43.279203 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:43.279187    2576 scope.go:117] "RemoveContainer" containerID="0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e"
Sep 04 22:29:43.280434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:43.280406    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:43.280499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:43.280477    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:43.280537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:43.280510    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:29:44.287139 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:44.287104    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" event={"ID":"97d3dcd1-941b-44bc-a405-9b1c774e84d4","Type":"ContainerStarted","Data":"d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1"}
Sep 04 22:29:45.023565 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:45.023519    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:45.023832 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:45.023804    2576 patch_prober.go:28] interesting pod/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 04 22:29:45.023932 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:45.023905    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 04 22:29:45.461286 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:45.461247    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:46.023506 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:46.023466    2576 patch_prober.go:28] interesting pod/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 04 22:29:46.023697 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:46.023529    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 04 22:29:47.023365 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:47.023296    2576 patch_prober.go:28] interesting pod/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 04 22:29:47.023775 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:47.023433    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 04 22:29:48.023642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:48.023605    2576 patch_prober.go:28] interesting pod/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 04 22:29:48.024110 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:48.023664    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 04 22:29:48.237473 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:48.237422    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:49.023086 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:49.023025    2576 patch_prober.go:28] interesting pod/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Sep 04 22:29:49.023310 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:49.023136    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Sep 04 22:29:50.027328 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:50.027293    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:50.027712 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:50.027556    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:50.028301 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:50.028285    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:29:51.786928 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.786898    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"]
Sep 04 22:29:51.791673 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.791655    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:51.795557 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.795530    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bc62f0\"/\"kube-root-ca.crt\""
Sep 04 22:29:51.795867 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.795848    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bc62f0\"/\"openshift-service-ca.crt\""
Sep 04 22:29:51.796103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.796084    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bc62f0\"/\"auth-test-model-bc62f0-kserve-self-signed-certs\""
Sep 04 22:29:51.798947 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.796300    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bc62f0\"/\"default-dockercfg-7pmj4\""
Sep 04 22:29:51.804257 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.804233    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"]
Sep 04 22:29:51.813263 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.813228    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:51.922184 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.922144    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-dshm\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:51.922342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.922208    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llhkm\" (UniqueName: \"kubernetes.io/projected/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-kube-api-access-llhkm\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:51.922342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.922255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-model-cache\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:51.922342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.922278    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tmp-dir\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:51.922342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.922336    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tls-certs\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:51.922503 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:51.922351    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-home\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.023637 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.023593    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-dshm\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.023839 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.023651    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llhkm\" (UniqueName: \"kubernetes.io/projected/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-kube-api-access-llhkm\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.023839 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.023672    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-model-cache\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.023839 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.023692    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tmp-dir\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.023839 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.023734    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tls-certs\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.023839 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.023756    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-home\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.024222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.024194    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tmp-dir\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.024222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.024202    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-model-cache\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.024371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.024225    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-home\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.026047 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.026026    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-dshm\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.026339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.026318    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tls-certs\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.034598 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.034568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llhkm\" (UniqueName: \"kubernetes.io/projected/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-kube-api-access-llhkm\") pod \"auth-test-model-bc62f0-kserve-76568549c5-8mzfz\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.107628 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.107524    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:29:52.384783 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:52.384735    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"]
Sep 04 22:29:52.387328 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:29:52.387294    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5aa5392_945f_4df7_bc4e_7d310c6acaa2.slice/crio-499003b35368ef911202fa418299b74e803c8452080ef355f0461bb75f26533b WatchSource:0}: Error finding container 499003b35368ef911202fa418299b74e803c8452080ef355f0461bb75f26533b: Status 404 returned error can't find the container with id 499003b35368ef911202fa418299b74e803c8452080ef355f0461bb75f26533b
Sep 04 22:29:53.339678 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:53.339633    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz" event={"ID":"e5aa5392-945f-4df7-bc4e-7d310c6acaa2","Type":"ContainerStarted","Data":"99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c"}
Sep 04 22:29:53.340185 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:53.339685    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz" event={"ID":"e5aa5392-945f-4df7-bc4e-7d310c6acaa2","Type":"ContainerStarted","Data":"499003b35368ef911202fa418299b74e803c8452080ef355f0461bb75f26533b"}
Sep 04 22:29:53.357250 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:53.357185    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz" podStartSLOduration=2.357167706 podStartE2EDuration="2.357167706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:29:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:29:53.356541499 +0000 UTC m=+1276.875264044" watchObservedRunningTime="2026-09-04 22:29:53.357167706 +0000 UTC m=+1276.875890240"
Sep 04 22:29:53.731275 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:53.731245    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:29:56.996896 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:29:56.996843    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:30:00.167896 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.167861    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq"]
Sep 04 22:30:00.171672 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.171654    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5"]
Sep 04 22:30:00.171787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.171767    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq"
Sep 04 22:30:00.174535 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.174513    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-hc7x4\""
Sep 04 22:30:00.175098 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.175072    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5"
Sep 04 22:30:00.182445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.182421    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq"]
Sep 04 22:30:00.184788 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.184751    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5"]
Sep 04 22:30:00.188931 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.188909    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq"
Sep 04 22:30:00.197066 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.197036    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5"
Sep 04 22:30:00.472498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.469772    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq"]
Sep 04 22:30:00.473604 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:30:00.473556    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715d23b7_fe71_4d06_bad4_1734489549d6.slice/crio-aadcd09044beef995b50142d0de8a2b12f7e8f73009b249e6d8350a5b20ef4c9 WatchSource:0}: Error finding container aadcd09044beef995b50142d0de8a2b12f7e8f73009b249e6d8350a5b20ef4c9: Status 404 returned error can't find the container with id aadcd09044beef995b50142d0de8a2b12f7e8f73009b249e6d8350a5b20ef4c9
Sep 04 22:30:00.486401 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:00.486376    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5"]
Sep 04 22:30:00.488408 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:30:00.488383    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd88177_c862_4c42_ab6e_100fc9e68ff0.slice/crio-b244cf9ae24d0feea56d5b931a3ee81ece2a6e4457ee3139c22d0584031d8bbb WatchSource:0}: Error finding container b244cf9ae24d0feea56d5b931a3ee81ece2a6e4457ee3139c22d0584031d8bbb: Status 404 returned error can't find the container with id b244cf9ae24d0feea56d5b931a3ee81ece2a6e4457ee3139c22d0584031d8bbb
Sep 04 22:30:01.385819 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:01.385780    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq" event={"ID":"715d23b7-fe71-4d06-bad4-1734489549d6","Type":"ContainerStarted","Data":"aadcd09044beef995b50142d0de8a2b12f7e8f73009b249e6d8350a5b20ef4c9"}
Sep 04 22:30:01.386973 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:01.386948    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5" event={"ID":"7bd88177-c862-4c42-ab6e-100fc9e68ff0","Type":"ContainerStarted","Data":"b244cf9ae24d0feea56d5b931a3ee81ece2a6e4457ee3139c22d0584031d8bbb"}
Sep 04 22:30:02.108298 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:02.107756    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:30:02.108298 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:02.107821    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:30:02.125234 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:02.125198    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:30:02.406526 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:02.406432    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:30:03.400105 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:03.400063    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5" event={"ID":"7bd88177-c862-4c42-ab6e-100fc9e68ff0","Type":"ContainerStarted","Data":"2fe59db8009e889514ea92fc69c3c172960a3afa16e0da0f3b702297d78a56fc"}
Sep 04 22:30:03.401655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:03.401623    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq" event={"ID":"715d23b7-fe71-4d06-bad4-1734489549d6","Type":"ContainerStarted","Data":"aecb0d69cef567e689fc521c4e9be203fa533ede68b4a7d84befb312b95ac223"}
Sep 04 22:30:03.417175 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:03.417122    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5" podStartSLOduration=0.845078198 podStartE2EDuration="3.417102101s" totalImagesPullingTime="2.572023903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:30:03.416767809 +0000 UTC m=+1286.935490346" watchObservedRunningTime="2026-09-04 22:30:03.417102101 +0000 UTC m=+1286.935824635"
Sep 04 22:30:03.433340 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:03.433292    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq" podStartSLOduration=0.851613415 podStartE2EDuration="3.433278334s" totalImagesPullingTime="2.581664919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:30:03.430938091 +0000 UTC m=+1286.949660625" watchObservedRunningTime="2026-09-04 22:30:03.433278334 +0000 UTC m=+1286.952000867"
Sep 04 22:30:04.408040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:04.407947    2576 generic.go:356] "Generic (PLEG): container finished" podID="7bd88177-c862-4c42-ab6e-100fc9e68ff0" containerID="2fe59db8009e889514ea92fc69c3c172960a3afa16e0da0f3b702297d78a56fc" exitCode=0
Sep 04 22:30:04.408246 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:04.408035    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5" event={"ID":"7bd88177-c862-4c42-ab6e-100fc9e68ff0","Type":"ContainerDied","Data":"2fe59db8009e889514ea92fc69c3c172960a3afa16e0da0f3b702297d78a56fc"}
Sep 04 22:30:04.409731 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:04.409705    2576 generic.go:356] "Generic (PLEG): container finished" podID="715d23b7-fe71-4d06-bad4-1734489549d6" containerID="aecb0d69cef567e689fc521c4e9be203fa533ede68b4a7d84befb312b95ac223" exitCode=0
Sep 04 22:30:04.409842 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:04.409733    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq" event={"ID":"715d23b7-fe71-4d06-bad4-1734489549d6","Type":"ContainerDied","Data":"aecb0d69cef567e689fc521c4e9be203fa533ede68b4a7d84befb312b95ac223"}
Sep 04 22:30:05.522985 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:05.522939    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:30:05.553106 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:05.553077    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq"
Sep 04 22:30:05.601892 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:05.601863    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5"
Sep 04 22:30:06.421446 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:06.421415    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq"
Sep 04 22:30:06.421446 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:06.421437    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq" event={"ID":"715d23b7-fe71-4d06-bad4-1734489549d6","Type":"ContainerDied","Data":"aadcd09044beef995b50142d0de8a2b12f7e8f73009b249e6d8350a5b20ef4c9"}
Sep 04 22:30:06.421705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:06.421484    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aadcd09044beef995b50142d0de8a2b12f7e8f73009b249e6d8350a5b20ef4c9"
Sep 04 22:30:06.422792 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:06.422776    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5"
Sep 04 22:30:06.422901 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:06.422773    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29809350-44nm5" event={"ID":"7bd88177-c862-4c42-ab6e-100fc9e68ff0","Type":"ContainerDied","Data":"b244cf9ae24d0feea56d5b931a3ee81ece2a6e4457ee3139c22d0584031d8bbb"}
Sep 04 22:30:06.422901 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:06.422897    2576 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b244cf9ae24d0feea56d5b931a3ee81ece2a6e4457ee3139c22d0584031d8bbb"
Sep 04 22:30:12.721573 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:12.721535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:30:19.198075 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:19.198018    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:30:33.894628 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.894595    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"]
Sep 04 22:30:33.895064 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.895036    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd88177-c862-4c42-ab6e-100fc9e68ff0" containerName="cleanup"
Sep 04 22:30:33.895127 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.895066    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd88177-c862-4c42-ab6e-100fc9e68ff0" containerName="cleanup"
Sep 04 22:30:33.895127 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.895081    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="715d23b7-fe71-4d06-bad4-1734489549d6" containerName="cleanup"
Sep 04 22:30:33.895127 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.895087    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="715d23b7-fe71-4d06-bad4-1734489549d6" containerName="cleanup"
Sep 04 22:30:33.895223 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.895157    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="715d23b7-fe71-4d06-bad4-1734489549d6" containerName="cleanup"
Sep 04 22:30:33.895223 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.895166    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd88177-c862-4c42-ab6e-100fc9e68ff0" containerName="cleanup"
Sep 04 22:30:33.903434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.903414    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:33.906794 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.906769    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"kube-root-ca.crt\""
Sep 04 22:30:33.907124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.906793    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"auth-test-model-bf86db-kserve-self-signed-certs\""
Sep 04 22:30:33.907124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.906808    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"default-dockercfg-nzfvk\""
Sep 04 22:30:33.907124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.906783    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"openshift-service-ca.crt\""
Sep 04 22:30:33.908144 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.908121    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"]
Sep 04 22:30:33.924035 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.924001    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:30:33.953203 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.953168    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vz86\" (UniqueName: \"kubernetes.io/projected/bef15946-2d3c-4a51-ae9c-00a5014d8685-kube-api-access-4vz86\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:33.953333 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.953207    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef15946-2d3c-4a51-ae9c-00a5014d8685-tls-certs\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:33.953333 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.953235    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-tmp-dir\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:33.953333 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.953318    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-home\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:33.953467 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.953350    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-model-cache\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:33.953467 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:33.953406    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-dshm\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.054455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.054417    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-home\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.054455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.054457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-model-cache\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.054731 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.054574    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-dshm\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.054731 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.054706    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vz86\" (UniqueName: \"kubernetes.io/projected/bef15946-2d3c-4a51-ae9c-00a5014d8685-kube-api-access-4vz86\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.054846 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.054735    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef15946-2d3c-4a51-ae9c-00a5014d8685-tls-certs\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.054846 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.054775    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-tmp-dir\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.054951 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.054904    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-home\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.054951 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.054919    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-model-cache\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.055153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.055127    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-tmp-dir\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.056995 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.056974    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-dshm\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.057204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.057187    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef15946-2d3c-4a51-ae9c-00a5014d8685-tls-certs\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.063877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.063853    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vz86\" (UniqueName: \"kubernetes.io/projected/bef15946-2d3c-4a51-ae9c-00a5014d8685-kube-api-access-4vz86\") pod \"auth-test-model-bf86db-kserve-68946f596c-kfgwd\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.095644 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.095605    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:30:34.215070 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.215029    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:34.494874 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.494847    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"]
Sep 04 22:30:34.496464 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:30:34.496427    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbef15946_2d3c_4a51_ae9c_00a5014d8685.slice/crio-7498a8e00cd1c256020dbc43a87932e0efe1d0876014d726b25ceba98aa9cb4c WatchSource:0}: Error finding container 7498a8e00cd1c256020dbc43a87932e0efe1d0876014d726b25ceba98aa9cb4c: Status 404 returned error can't find the container with id 7498a8e00cd1c256020dbc43a87932e0efe1d0876014d726b25ceba98aa9cb4c
Sep 04 22:30:34.583229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.583196    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd" event={"ID":"bef15946-2d3c-4a51-ae9c-00a5014d8685","Type":"ContainerStarted","Data":"085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476"}
Sep 04 22:30:34.583382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.583253    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd" event={"ID":"bef15946-2d3c-4a51-ae9c-00a5014d8685","Type":"ContainerStarted","Data":"7498a8e00cd1c256020dbc43a87932e0efe1d0876014d726b25ceba98aa9cb4c"}
Sep 04 22:30:34.601318 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:34.601261    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd" podStartSLOduration=1.601243255 podStartE2EDuration="1.601243255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:30:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:30:34.600128553 +0000 UTC m=+1318.118851088" watchObservedRunningTime="2026-09-04 22:30:34.601243255 +0000 UTC m=+1318.119965824"
Sep 04 22:30:38.538828 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:38.538790    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:30:44.215226 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:44.215180    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:44.215226 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:44.215232    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:44.227825 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:44.227796    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:44.649645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:44.649554    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:30:53.695786 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:53.695746    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:30:54.002402 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:54.002365    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:30:54.067257 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:54.067218    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:30:54.170351 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:54.170315    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:30:54.249882 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:54.249843    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:30:57.730548 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:30:57.730503    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:31:02.808008 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:31:02.807915    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:31:06.540598 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:31:06.540555    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:31:11.698242 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:31:11.698205    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:31:28.799136 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:31:28.799096    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:31:40.118217 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:31:40.118178    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:31:40.455315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:31:40.455272    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:31:43.700124 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:31:43.700077    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:31:46.685728 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:31:46.685691    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:00.299866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:00.299825    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:32:01.498285 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:01.498245    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:01.599729 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:01.599685    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:01.761385 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:01.761296    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"]
Sep 04 22:32:01.761573 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:01.761525    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy" containerID="cri-o://d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1" gracePeriod=30
Sep 04 22:32:01.787160 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:01.787100    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:03.122486 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:03.122452    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:03.223633 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:03.223593    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:03.325320 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:03.325285    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:04.007297 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:04.007257    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:04.109480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:04.109443    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:04.206733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:04.206702    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:05.030333 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:05.030293    2576 patch_prober.go:28] interesting pod/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 22:32:05.030494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:05.030374    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:32:05.599253 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:05.599199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:07.005009 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.004978    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:32:07.079192 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.079106    2576 generic.go:356] "Generic (PLEG): container finished" podID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerID="d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1" exitCode=0
Sep 04 22:32:07.079192 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.079157    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"
Sep 04 22:32:07.079400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.079188    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" event={"ID":"97d3dcd1-941b-44bc-a405-9b1c774e84d4","Type":"ContainerDied","Data":"d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1"}
Sep 04 22:32:07.079400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.079239    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp" event={"ID":"97d3dcd1-941b-44bc-a405-9b1c774e84d4","Type":"ContainerDied","Data":"3bf260f93eb2fc9ac5611f377933abc45825886c0328c2708a061b8f015709c8"}
Sep 04 22:32:07.079400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.079267    2576 scope.go:117] "RemoveContainer" containerID="d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1"
Sep 04 22:32:07.088264 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.088247    2576 scope.go:117] "RemoveContainer" containerID="0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e"
Sep 04 22:32:07.096891 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.096869    2576 scope.go:117] "RemoveContainer" containerID="d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1"
Sep 04 22:32:07.097271 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:32:07.097252    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1\": container with ID starting with d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1 not found: ID does not exist" containerID="d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1"
Sep 04 22:32:07.097322 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.097281    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1"} err="failed to get container status \"d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1\": rpc error: code = NotFound desc = could not find container \"d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1\": container with ID starting with d4f191df75a371b4c47ee2b581d1dbdb131513520ae8e6fab0078d1e51dc05e1 not found: ID does not exist"
Sep 04 22:32:07.097322 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.097299    2576 scope.go:117] "RemoveContainer" containerID="0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e"
Sep 04 22:32:07.097546 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:32:07.097527    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e\": container with ID starting with 0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e not found: ID does not exist" containerID="0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e"
Sep 04 22:32:07.097587 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.097554    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e"} err="failed to get container status \"0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e\": rpc error: code = NotFound desc = could not find container \"0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e\": container with ID starting with 0798143e99ec9acefede7b62340efc9ec4d860ab96ca49c0adb9e563b9fb859e not found: ID does not exist"
Sep 04 22:32:07.112944 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.112917    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-credential-socket\") pod \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") "
Sep 04 22:32:07.113096 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.112976    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrbbc\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-kube-api-access-lrbbc\") pod \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") "
Sep 04 22:32:07.113096 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113049    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istiod-ca-cert\") pod \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") "
Sep 04 22:32:07.113196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113104    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-data\") pod \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") "
Sep 04 22:32:07.113196 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113157    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-token\") pod \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") "
Sep 04 22:32:07.113281 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113194    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-socket\") pod \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") "
Sep 04 22:32:07.113281 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113236    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-envoy\") pod \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") "
Sep 04 22:32:07.113281 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113257    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "97d3dcd1-941b-44bc-a405-9b1c774e84d4" (UID: "97d3dcd1-941b-44bc-a405-9b1c774e84d4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:32:07.113433 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113279    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-certs\") pod \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") "
Sep 04 22:32:07.113433 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113320    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-podinfo\") pod \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\" (UID: \"97d3dcd1-941b-44bc-a405-9b1c774e84d4\") "
Sep 04 22:32:07.113540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113468    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-data" (OuterVolumeSpecName: "istio-data") pod "97d3dcd1-941b-44bc-a405-9b1c774e84d4" (UID: "97d3dcd1-941b-44bc-a405-9b1c774e84d4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:32:07.113540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113480    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "97d3dcd1-941b-44bc-a405-9b1c774e84d4" (UID: "97d3dcd1-941b-44bc-a405-9b1c774e84d4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:32:07.113653 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113604    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "97d3dcd1-941b-44bc-a405-9b1c774e84d4" (UID: "97d3dcd1-941b-44bc-a405-9b1c774e84d4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:32:07.113729 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113710    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "97d3dcd1-941b-44bc-a405-9b1c774e84d4" (UID: "97d3dcd1-941b-44bc-a405-9b1c774e84d4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:32:07.113796 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113777    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:32:07.113860 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113802    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-credential-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:32:07.113860 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113817    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-data\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:32:07.113860 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.113829    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-workload-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:32:07.115428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.115395    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-token" (OuterVolumeSpecName: "istio-token") pod "97d3dcd1-941b-44bc-a405-9b1c774e84d4" (UID: "97d3dcd1-941b-44bc-a405-9b1c774e84d4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:32:07.115549 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.115529    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "97d3dcd1-941b-44bc-a405-9b1c774e84d4" (UID: "97d3dcd1-941b-44bc-a405-9b1c774e84d4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:32:07.115663 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.115646    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-kube-api-access-lrbbc" (OuterVolumeSpecName: "kube-api-access-lrbbc") pod "97d3dcd1-941b-44bc-a405-9b1c774e84d4" (UID: "97d3dcd1-941b-44bc-a405-9b1c774e84d4"). InnerVolumeSpecName "kube-api-access-lrbbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:32:07.115722 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.115655    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "97d3dcd1-941b-44bc-a405-9b1c774e84d4" (UID: "97d3dcd1-941b-44bc-a405-9b1c774e84d4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 22:32:07.214735 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.214690    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrbbc\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-kube-api-access-lrbbc\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:32:07.214735 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.214729    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istiod-ca-cert\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:32:07.214735 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.214742    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-token\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:32:07.214735 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.214750    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-envoy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:32:07.215020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.214758    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97d3dcd1-941b-44bc-a405-9b1c774e84d4-istio-podinfo\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:32:07.407897 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.407862    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"]
Sep 04 22:32:07.415746 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:07.415713    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-3949cd16-openshift-default-b66ddc8b4-z6thp"]
Sep 04 22:32:08.918330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:08.918299    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" path="/var/lib/kubelet/pods/97d3dcd1-941b-44bc-a405-9b1c774e84d4/volumes"
Sep 04 22:32:11.865120 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:11.865076    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:11.941235 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:11.941198    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:12.565855 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:12.565819    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:17.397244 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:17.397202    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:32:18.632379 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:18.632337    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:18.732074 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:18.732022    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:18.828400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:18.828373    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:29.351159 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:29.351040    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:29.578601 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:29.578566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:32.999479 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:32.999440    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:32:46.010136 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:46.010092    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"]
Sep 04 22:32:46.075918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:46.075882    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:47.093907 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:47.093867    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"]
Sep 04 22:32:47.125655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:47.125620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:48.495383 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:48.495346    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:32:50.554297 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:50.554249    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:51.642322 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:51.642265    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:57.666540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:57.666497    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:57.760503 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:57.760466    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:58.052085 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:58.052038    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:58.415716 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:58.415617    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:32:58.598497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:32:58.598464    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:01.030253 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.030207    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz" podUID="e5aa5392-945f-4df7-bc4e-7d310c6acaa2" containerName="main" containerID="cri-o://99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c" gracePeriod=45
Sep 04 22:33:01.281804 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.281739    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:33:01.366209 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366180    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-dshm\") pod \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") "
Sep 04 22:33:01.366413 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366250    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llhkm\" (UniqueName: \"kubernetes.io/projected/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-kube-api-access-llhkm\") pod \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") "
Sep 04 22:33:01.366413 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366302    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-model-cache\") pod \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") "
Sep 04 22:33:01.366413 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366400    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tls-certs\") pod \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") "
Sep 04 22:33:01.366585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366427    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-home\") pod \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") "
Sep 04 22:33:01.366585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366460    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tmp-dir\") pod \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\" (UID: \"e5aa5392-945f-4df7-bc4e-7d310c6acaa2\") "
Sep 04 22:33:01.366585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366549    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-model-cache" (OuterVolumeSpecName: "model-cache") pod "e5aa5392-945f-4df7-bc4e-7d310c6acaa2" (UID: "e5aa5392-945f-4df7-bc4e-7d310c6acaa2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:33:01.366758 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366718    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-home" (OuterVolumeSpecName: "home") pod "e5aa5392-945f-4df7-bc4e-7d310c6acaa2" (UID: "e5aa5392-945f-4df7-bc4e-7d310c6acaa2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:33:01.366758 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366728    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e5aa5392-945f-4df7-bc4e-7d310c6acaa2" (UID: "e5aa5392-945f-4df7-bc4e-7d310c6acaa2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:33:01.366872 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366854    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:01.366929 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366874    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:01.366929 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.366888    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:01.368628 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.368598    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e5aa5392-945f-4df7-bc4e-7d310c6acaa2" (UID: "e5aa5392-945f-4df7-bc4e-7d310c6acaa2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:33:01.368628 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.368613    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-kube-api-access-llhkm" (OuterVolumeSpecName: "kube-api-access-llhkm") pod "e5aa5392-945f-4df7-bc4e-7d310c6acaa2" (UID: "e5aa5392-945f-4df7-bc4e-7d310c6acaa2"). InnerVolumeSpecName "kube-api-access-llhkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:33:01.368834 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.368724    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-dshm" (OuterVolumeSpecName: "dshm") pod "e5aa5392-945f-4df7-bc4e-7d310c6acaa2" (UID: "e5aa5392-945f-4df7-bc4e-7d310c6acaa2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:33:01.371288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.371263    2576 generic.go:356] "Generic (PLEG): container finished" podID="e5aa5392-945f-4df7-bc4e-7d310c6acaa2" containerID="99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c" exitCode=0
Sep 04 22:33:01.371421 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.371314    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"
Sep 04 22:33:01.371421 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.371344    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz" event={"ID":"e5aa5392-945f-4df7-bc4e-7d310c6acaa2","Type":"ContainerDied","Data":"99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c"}
Sep 04 22:33:01.371421 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.371386    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz" event={"ID":"e5aa5392-945f-4df7-bc4e-7d310c6acaa2","Type":"ContainerDied","Data":"499003b35368ef911202fa418299b74e803c8452080ef355f0461bb75f26533b"}
Sep 04 22:33:01.371421 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.371404    2576 scope.go:117] "RemoveContainer" containerID="99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c"
Sep 04 22:33:01.389875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.389847    2576 scope.go:117] "RemoveContainer" containerID="99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c"
Sep 04 22:33:01.390234 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:33:01.390215    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c\": container with ID starting with 99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c not found: ID does not exist" containerID="99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c"
Sep 04 22:33:01.390301 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.390243    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c"} err="failed to get container status \"99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c\": rpc error: code = NotFound desc = could not find container \"99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c\": container with ID starting with 99d6717abcb6be4afcbb42b76071a23be110b802dae28a9af7c009c957b7d60c not found: ID does not exist"
Sep 04 22:33:01.404142 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.404118    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"]
Sep 04 22:33:01.415108 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.415084    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/auth-test-model-bc62f0-kserve-76568549c5-8mzfz"]
Sep 04 22:33:01.467665 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.467630    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llhkm\" (UniqueName: \"kubernetes.io/projected/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-kube-api-access-llhkm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:01.467665 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.467664    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:01.467848 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:01.467675    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e5aa5392-945f-4df7-bc4e-7d310c6acaa2-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:02.113844 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.113801    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd" podUID="bef15946-2d3c-4a51-ae9c-00a5014d8685" containerName="main" containerID="cri-o://085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476" gracePeriod=45
Sep 04 22:33:02.144585 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.144536    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:02.247202 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.247165    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:02.366407 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.366337    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:33:02.379390 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.379358    2576 generic.go:356] "Generic (PLEG): container finished" podID="bef15946-2d3c-4a51-ae9c-00a5014d8685" containerID="085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476" exitCode=0
Sep 04 22:33:02.379584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.379407    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"
Sep 04 22:33:02.379584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.379451    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd" event={"ID":"bef15946-2d3c-4a51-ae9c-00a5014d8685","Type":"ContainerDied","Data":"085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476"}
Sep 04 22:33:02.379584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.379492    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd" event={"ID":"bef15946-2d3c-4a51-ae9c-00a5014d8685","Type":"ContainerDied","Data":"7498a8e00cd1c256020dbc43a87932e0efe1d0876014d726b25ceba98aa9cb4c"}
Sep 04 22:33:02.379584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.379514    2576 scope.go:117] "RemoveContainer" containerID="085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476"
Sep 04 22:33:02.388121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.388100    2576 scope.go:117] "RemoveContainer" containerID="085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476"
Sep 04 22:33:02.388476 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:33:02.388456    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476\": container with ID starting with 085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476 not found: ID does not exist" containerID="085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476"
Sep 04 22:33:02.388534 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.388484    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476"} err="failed to get container status \"085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476\": rpc error: code = NotFound desc = could not find container \"085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476\": container with ID starting with 085d2b9b3a4e2c6c5225e8f98b0d480c71c8fcdeb08d108166a71f8f01246476 not found: ID does not exist"
Sep 04 22:33:02.477501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.477463    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-tmp-dir\") pod \"bef15946-2d3c-4a51-ae9c-00a5014d8685\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") "
Sep 04 22:33:02.477688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.477525    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-dshm\") pod \"bef15946-2d3c-4a51-ae9c-00a5014d8685\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") "
Sep 04 22:33:02.477688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.477553    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vz86\" (UniqueName: \"kubernetes.io/projected/bef15946-2d3c-4a51-ae9c-00a5014d8685-kube-api-access-4vz86\") pod \"bef15946-2d3c-4a51-ae9c-00a5014d8685\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") "
Sep 04 22:33:02.477688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.477643    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-home\") pod \"bef15946-2d3c-4a51-ae9c-00a5014d8685\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") "
Sep 04 22:33:02.477894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.477691    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-model-cache\") pod \"bef15946-2d3c-4a51-ae9c-00a5014d8685\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") "
Sep 04 22:33:02.477894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.477752    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef15946-2d3c-4a51-ae9c-00a5014d8685-tls-certs\") pod \"bef15946-2d3c-4a51-ae9c-00a5014d8685\" (UID: \"bef15946-2d3c-4a51-ae9c-00a5014d8685\") "
Sep 04 22:33:02.477894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.477849    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bef15946-2d3c-4a51-ae9c-00a5014d8685" (UID: "bef15946-2d3c-4a51-ae9c-00a5014d8685"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:33:02.478096 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.477934    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-home" (OuterVolumeSpecName: "home") pod "bef15946-2d3c-4a51-ae9c-00a5014d8685" (UID: "bef15946-2d3c-4a51-ae9c-00a5014d8685"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:33:02.478096 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.478019    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-model-cache" (OuterVolumeSpecName: "model-cache") pod "bef15946-2d3c-4a51-ae9c-00a5014d8685" (UID: "bef15946-2d3c-4a51-ae9c-00a5014d8685"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:33:02.478213 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.478164    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:02.478213 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.478181    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:02.478213 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.478204    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:02.479950 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.479926    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-dshm" (OuterVolumeSpecName: "dshm") pod "bef15946-2d3c-4a51-ae9c-00a5014d8685" (UID: "bef15946-2d3c-4a51-ae9c-00a5014d8685"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:33:02.480068 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.479929    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bef15946-2d3c-4a51-ae9c-00a5014d8685-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bef15946-2d3c-4a51-ae9c-00a5014d8685" (UID: "bef15946-2d3c-4a51-ae9c-00a5014d8685"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:33:02.480108 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.480045    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bef15946-2d3c-4a51-ae9c-00a5014d8685-kube-api-access-4vz86" (OuterVolumeSpecName: "kube-api-access-4vz86") pod "bef15946-2d3c-4a51-ae9c-00a5014d8685" (UID: "bef15946-2d3c-4a51-ae9c-00a5014d8685"). InnerVolumeSpecName "kube-api-access-4vz86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:33:02.578916 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.578873    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bef15946-2d3c-4a51-ae9c-00a5014d8685-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:02.578916 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.578908    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bef15946-2d3c-4a51-ae9c-00a5014d8685-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:02.578916 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.578917    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vz86\" (UniqueName: \"kubernetes.io/projected/bef15946-2d3c-4a51-ae9c-00a5014d8685-kube-api-access-4vz86\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:33:02.715087 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.715029    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"]
Sep 04 22:33:02.731874 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.731843    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/auth-test-model-bf86db-kserve-68946f596c-kfgwd"]
Sep 04 22:33:02.918562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.918518    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bef15946-2d3c-4a51-ae9c-00a5014d8685" path="/var/lib/kubelet/pods/bef15946-2d3c-4a51-ae9c-00a5014d8685/volumes"
Sep 04 22:33:02.919030 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:02.919009    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5aa5392-945f-4df7-bc4e-7d310c6acaa2" path="/var/lib/kubelet/pods/e5aa5392-945f-4df7-bc4e-7d310c6acaa2/volumes"
Sep 04 22:33:03.313958 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.313919    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"]
Sep 04 22:33:03.314631 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314607    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5aa5392-945f-4df7-bc4e-7d310c6acaa2" containerName="main"
Sep 04 22:33:03.314710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314635    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5aa5392-945f-4df7-bc4e-7d310c6acaa2" containerName="main"
Sep 04 22:33:03.314710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314659    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy"
Sep 04 22:33:03.314710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314668    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy"
Sep 04 22:33:03.314710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314682    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bef15946-2d3c-4a51-ae9c-00a5014d8685" containerName="main"
Sep 04 22:33:03.314710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314691    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bef15946-2d3c-4a51-ae9c-00a5014d8685" containerName="main"
Sep 04 22:33:03.314866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314716    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy"
Sep 04 22:33:03.314866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314724    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy"
Sep 04 22:33:03.314866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314849    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy"
Sep 04 22:33:03.314981 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314867    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bef15946-2d3c-4a51-ae9c-00a5014d8685" containerName="main"
Sep 04 22:33:03.314981 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.314878    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5aa5392-945f-4df7-bc4e-7d310c6acaa2" containerName="main"
Sep 04 22:33:03.320159 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.320137    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.322842 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.322812    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b852ba-serving-cert\""
Sep 04 22:33:03.331969 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.331940    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"]
Sep 04 22:33:03.358588 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.358554    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:03.460275 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.460239    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:03.489595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.489551    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-openshift-service-ca\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.489811 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.489607    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxcm\" (UniqueName: \"kubernetes.io/projected/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-kube-api-access-skxcm\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.489811 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.489723    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-metrics-certs\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.489946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.489890    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-maas-api-tls\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.590883 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.590782    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-openshift-service-ca\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.590883 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.590832    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxcm\" (UniqueName: \"kubernetes.io/projected/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-kube-api-access-skxcm\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.590883 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.590881    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-metrics-certs\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.591204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.590955    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-maas-api-tls\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.591204 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:33:03.591084    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-serving-cert: secret "maas-api-e2e-api-a-b852ba-serving-cert" not found
Sep 04 22:33:03.591204 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:33:03.591142    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-maas-api-tls podName:edf8bf1d-f21e-4fd6-bb64-0c70469eddf4 nodeName:}" failed. No retries permitted until 2026-09-04 22:33:04.091124708 +0000 UTC m=+1467.609847236 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-maas-api-tls") pod "maas-api-e2e-api-a-b852ba-69c94db677-v6zsh" (UID: "edf8bf1d-f21e-4fd6-bb64-0c70469eddf4") : secret "maas-api-e2e-api-a-b852ba-serving-cert" not found
Sep 04 22:33:03.591507 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.591484    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-openshift-service-ca\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.593525 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.593502    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-metrics-certs\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.605624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.605592    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxcm\" (UniqueName: \"kubernetes.io/projected/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-kube-api-access-skxcm\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:03.758633 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:03.758599    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:04.096008 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:04.095973    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-maas-api-tls\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:04.098382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:04.098356    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-maas-api-tls\") pod \"maas-api-e2e-api-a-b852ba-69c94db677-v6zsh\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:04.118688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:04.118646    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:04.220165 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:04.220129    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:04.234941 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:04.234907    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:04.518420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:04.518386    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"]
Sep 04 22:33:04.519886 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:33:04.519862    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf8bf1d_f21e_4fd6_bb64_0c70469eddf4.slice/crio-eceaac90c1e0dcbd0b469257beb77a11e7eb58c1403bd6571b1a0c7f0552fd47 WatchSource:0}: Error finding container eceaac90c1e0dcbd0b469257beb77a11e7eb58c1403bd6571b1a0c7f0552fd47: Status 404 returned error can't find the container with id eceaac90c1e0dcbd0b469257beb77a11e7eb58c1403bd6571b1a0c7f0552fd47
Sep 04 22:33:04.521310 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:04.521292    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 22:33:04.618407 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:04.618369    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:33:05.399969 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:05.399928    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh" event={"ID":"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4","Type":"ContainerStarted","Data":"78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57"}
Sep 04 22:33:05.400190 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:05.399988    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh" event={"ID":"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4","Type":"ContainerStarted","Data":"eceaac90c1e0dcbd0b469257beb77a11e7eb58c1403bd6571b1a0c7f0552fd47"}
Sep 04 22:33:05.400190 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:05.400009    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:05.426518 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:05.426459    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh" podStartSLOduration=2.055261975 podStartE2EDuration="2.426440663s" totalImagesPullingTime="371.178688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:33:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:33:05.425337101 +0000 UTC m=+1468.944059647" watchObservedRunningTime="2026-09-04 22:33:05.426440663 +0000 UTC m=+1468.945163195"
Sep 04 22:33:11.411868 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:11.411836    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:33:21.667424 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:21.667372    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:33:23.228696 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.228655    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"]
Sep 04 22:33:23.229188 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.229173    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="97d3dcd1-941b-44bc-a405-9b1c774e84d4" containerName="istio-proxy"
Sep 04 22:33:23.232467 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.232451    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.235174 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.235153    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"sub-test-model-bf86db-kserve-self-signed-certs\""
Sep 04 22:33:23.236277 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.236255    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"openshift-service-ca.crt\""
Sep 04 22:33:23.236391 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.236302    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"default-dockercfg-nzfvk\""
Sep 04 22:33:23.236391 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.236358    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"kube-root-ca.crt\""
Sep 04 22:33:23.244398 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.244375    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"]
Sep 04 22:33:23.290469 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.290433    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-dshm\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.290625 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.290480    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-tmp-dir\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.290625 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.290511    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f550c80-de80-4a5c-88b8-0d58542852c3-tls-certs\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.290625 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.290570    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-model-cache\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.290625 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.290621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-home\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.290772 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.290663    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cplw9\" (UniqueName: \"kubernetes.io/projected/8f550c80-de80-4a5c-88b8-0d58542852c3-kube-api-access-cplw9\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.391922 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.391886    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-dshm\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.392134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.391936    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-tmp-dir\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.392134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.391963    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f550c80-de80-4a5c-88b8-0d58542852c3-tls-certs\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.392134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.391989    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-model-cache\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.392134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.392032    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-home\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.392134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.392083    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cplw9\" (UniqueName: \"kubernetes.io/projected/8f550c80-de80-4a5c-88b8-0d58542852c3-kube-api-access-cplw9\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.392429 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.392400    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-tmp-dir\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.392429 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.392419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-model-cache\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.392517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.392465    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-home\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.394398 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.394368    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-dshm\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.394666 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.394640    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f550c80-de80-4a5c-88b8-0d58542852c3-tls-certs\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.408690 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.408666    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cplw9\" (UniqueName: \"kubernetes.io/projected/8f550c80-de80-4a5c-88b8-0d58542852c3-kube-api-access-cplw9\") pod \"sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.430866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.430832    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:23.542261 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.542167    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:23.816679 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:23.816651    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"]
Sep 04 22:33:23.818613 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:33:23.818586    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8f550c80_de80_4a5c_88b8_0d58542852c3.slice/crio-abe2fae305b72bbd3bb4685bb4808f6c057523e26d10be91fff07736644a79ad WatchSource:0}: Error finding container abe2fae305b72bbd3bb4685bb4808f6c057523e26d10be91fff07736644a79ad: Status 404 returned error can't find the container with id abe2fae305b72bbd3bb4685bb4808f6c057523e26d10be91fff07736644a79ad
Sep 04 22:33:24.504897 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:24.504858    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk" event={"ID":"8f550c80-de80-4a5c-88b8-0d58542852c3","Type":"ContainerStarted","Data":"9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16"}
Sep 04 22:33:24.504897 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:24.504897    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk" event={"ID":"8f550c80-de80-4a5c-88b8-0d58542852c3","Type":"ContainerStarted","Data":"abe2fae305b72bbd3bb4685bb4808f6c057523e26d10be91fff07736644a79ad"}
Sep 04 22:33:24.530239 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:24.530184    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk" podStartSLOduration=1.5301677919999999 podStartE2EDuration="1.530167792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:33:24.529707845 +0000 UTC m=+1488.048430379" watchObservedRunningTime="2026-09-04 22:33:24.530167792 +0000 UTC m=+1488.048890325"
Sep 04 22:33:27.147675 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:27.147628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:27.622642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:27.622603    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:28.032708 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:28.032671    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:28.234521 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:28.234480    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:31.274739 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:31.274702    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:31.376154 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:31.376111    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:32.779300 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:32.779251    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:32.876961 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:32.876927    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:33.277262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:33.277218    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:33.367215 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:33.367179    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:33.542361 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:33.542276    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:33.542361 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:33.542324    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:33.556515 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:33.556477    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:33.567786 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:33.567756    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:33:36.984070 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:36.984019    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:33:36.985932 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:36.985909    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:33:37.817582 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:37.817543    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:33:40.911729 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:40.911694    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:33:55.245908 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:33:55.245872    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:01.440595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:01.440506    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:04.918726 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:04.918690    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:05.228891 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:05.228852    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:34:08.224514 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:08.224476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:09.603044 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:09.603009    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:09.785184 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:09.785137    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7743-8f0b21"]
Sep 04 22:34:09.789073 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:09.789034    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:09.792221 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:09.792198    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-dlrbm\""
Sep 04 22:34:09.802946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:09.802917    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7743-8f0b21"]
Sep 04 22:34:09.865766 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:09.865686    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m58zv\" (UniqueName: \"kubernetes.io/projected/7603fc96-607b-4466-b9ae-057042a24ac3-kube-api-access-m58zv\") pod \"mt-curl-7743-8f0b21\" (UID: \"7603fc96-607b-4466-b9ae-057042a24ac3\") " pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:09.967387 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:09.967342    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m58zv\" (UniqueName: \"kubernetes.io/projected/7603fc96-607b-4466-b9ae-057042a24ac3-kube-api-access-m58zv\") pod \"mt-curl-7743-8f0b21\" (UID: \"7603fc96-607b-4466-b9ae-057042a24ac3\") " pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:09.991845 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:09.991811    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m58zv\" (UniqueName: \"kubernetes.io/projected/7603fc96-607b-4466-b9ae-057042a24ac3-kube-api-access-m58zv\") pod \"mt-curl-7743-8f0b21\" (UID: \"7603fc96-607b-4466-b9ae-057042a24ac3\") " pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:10.098026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:10.097984    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:10.392759 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:10.392734    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7743-8f0b21"]
Sep 04 22:34:10.394170 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:34:10.394138    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7603fc96_607b_4466_b9ae_057042a24ac3.slice/crio-6f92d747c3793f92eaeaa9cfd8e8605d23196f3b5b6042a2911ae01f502e8a3b WatchSource:0}: Error finding container 6f92d747c3793f92eaeaa9cfd8e8605d23196f3b5b6042a2911ae01f502e8a3b: Status 404 returned error can't find the container with id 6f92d747c3793f92eaeaa9cfd8e8605d23196f3b5b6042a2911ae01f502e8a3b
Sep 04 22:34:10.751989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:10.751946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7743-8f0b21" event={"ID":"7603fc96-607b-4466-b9ae-057042a24ac3","Type":"ContainerStarted","Data":"f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670"}
Sep 04 22:34:10.751989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:10.751992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7743-8f0b21" event={"ID":"7603fc96-607b-4466-b9ae-057042a24ac3","Type":"ContainerStarted","Data":"6f92d747c3793f92eaeaa9cfd8e8605d23196f3b5b6042a2911ae01f502e8a3b"}
Sep 04 22:34:10.788314 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:10.788261    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/mt-curl-7743-8f0b21" podStartSLOduration=1.788244775 podStartE2EDuration="1.788244775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:34:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:34:10.787236612 +0000 UTC m=+1534.305959145" watchObservedRunningTime="2026-09-04 22:34:10.788244775 +0000 UTC m=+1534.306967307"
Sep 04 22:34:11.554249 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.554163    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7743-8f0b21"]
Sep 04 22:34:11.557372 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.557218    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7743-8f0b21"]
Sep 04 22:34:11.757897 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:11.757866    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/mt-curl-7743-8f0b21" container="mt-curl-7743-8f0b21"
Sep 04 22:34:11.807396 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.807325    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7743-8dfaea"]
Sep 04 22:34:11.807751 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.807738    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7603fc96-607b-4466-b9ae-057042a24ac3" containerName="mt-curl-7743-8f0b21"
Sep 04 22:34:11.807802 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.807752    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="7603fc96-607b-4466-b9ae-057042a24ac3" containerName="mt-curl-7743-8f0b21"
Sep 04 22:34:11.807850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.807839    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="7603fc96-607b-4466-b9ae-057042a24ac3" containerName="mt-curl-7743-8f0b21"
Sep 04 22:34:11.811119 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.811103    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:11.823368 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.823341    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7743-8dfaea"]
Sep 04 22:34:11.887547 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.887504    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vkk\" (UniqueName: \"kubernetes.io/projected/be70e249-56c9-44ea-805e-6d3131bd3bfa-kube-api-access-t8vkk\") pod \"mt-curl-7743-8dfaea\" (UID: \"be70e249-56c9-44ea-805e-6d3131bd3bfa\") " pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:11.989118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:11.989073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vkk\" (UniqueName: \"kubernetes.io/projected/be70e249-56c9-44ea-805e-6d3131bd3bfa-kube-api-access-t8vkk\") pod \"mt-curl-7743-8dfaea\" (UID: \"be70e249-56c9-44ea-805e-6d3131bd3bfa\") " pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:12.005326 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:12.005297    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vkk\" (UniqueName: \"kubernetes.io/projected/be70e249-56c9-44ea-805e-6d3131bd3bfa-kube-api-access-t8vkk\") pod \"mt-curl-7743-8dfaea\" (UID: \"be70e249-56c9-44ea-805e-6d3131bd3bfa\") " pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:12.120580 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:12.120479    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:12.408358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:12.408327    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7743-8dfaea"]
Sep 04 22:34:12.408872 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:34:12.408847    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe70e249_56c9_44ea_805e_6d3131bd3bfa.slice/crio-b0025d605d36634cca498d4c9b01b0727115ecce1d1082bb18b45a93deb42008 WatchSource:0}: Error finding container b0025d605d36634cca498d4c9b01b0727115ecce1d1082bb18b45a93deb42008: Status 404 returned error can't find the container with id b0025d605d36634cca498d4c9b01b0727115ecce1d1082bb18b45a93deb42008
Sep 04 22:34:12.764632 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:12.764598    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7743-8dfaea" event={"ID":"be70e249-56c9-44ea-805e-6d3131bd3bfa","Type":"ContainerStarted","Data":"4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc"}
Sep 04 22:34:12.765039 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:12.764640    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7743-8dfaea" event={"ID":"be70e249-56c9-44ea-805e-6d3131bd3bfa","Type":"ContainerStarted","Data":"b0025d605d36634cca498d4c9b01b0727115ecce1d1082bb18b45a93deb42008"}
Sep 04 22:34:12.765039 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:12.764739    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/mt-curl-7743-8f0b21" podUID="7603fc96-607b-4466-b9ae-057042a24ac3" containerName="mt-curl-7743-8f0b21" containerID="cri-o://f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670" gracePeriod=2
Sep 04 22:34:12.790484 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:12.790429    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/mt-curl-7743-8dfaea" podStartSLOduration=1.790411589 podStartE2EDuration="1.790411589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:34:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:34:12.78981468 +0000 UTC m=+1536.308537213" watchObservedRunningTime="2026-09-04 22:34:12.790411589 +0000 UTC m=+1536.309134157"
Sep 04 22:34:12.792342 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:12.792310    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7743-8f0b21\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="7603fc96-607b-4466-b9ae-057042a24ac3" pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:13.651300 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:13.651208    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7743-8dfaea"]
Sep 04 22:34:13.670143 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:13.669993    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7743-8dfaea"]
Sep 04 22:34:13.769859 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:13.769825    2576 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/mt-curl-7743-8dfaea" container="mt-curl-7743-8dfaea"
Sep 04 22:34:14.775358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:14.775316    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/mt-curl-7743-8dfaea" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" containerName="mt-curl-7743-8dfaea" containerID="cri-o://4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc" gracePeriod=2
Sep 04 22:34:14.783877 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:14.783843    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7743-8dfaea\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:14.965517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:14.965492    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:15.021463 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:15.021347    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:15.122868 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:15.122823    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m58zv\" (UniqueName: \"kubernetes.io/projected/7603fc96-607b-4466-b9ae-057042a24ac3-kube-api-access-m58zv\") pod \"7603fc96-607b-4466-b9ae-057042a24ac3\" (UID: \"7603fc96-607b-4466-b9ae-057042a24ac3\") "
Sep 04 22:34:15.124948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:15.124918    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7603fc96-607b-4466-b9ae-057042a24ac3-kube-api-access-m58zv" (OuterVolumeSpecName: "kube-api-access-m58zv") pod "7603fc96-607b-4466-b9ae-057042a24ac3" (UID: "7603fc96-607b-4466-b9ae-057042a24ac3"). InnerVolumeSpecName "kube-api-access-m58zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:34:15.224153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:15.224123    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m58zv\" (UniqueName: \"kubernetes.io/projected/7603fc96-607b-4466-b9ae-057042a24ac3-kube-api-access-m58zv\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:15.781841 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:15.781801    2576 generic.go:356] "Generic (PLEG): container finished" podID="7603fc96-607b-4466-b9ae-057042a24ac3" containerID="f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670" exitCode=137
Sep 04 22:34:15.781841 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:15.781846    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:15.782346 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:15.781906    2576 scope.go:117] "RemoveContainer" containerID="f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670"
Sep 04 22:34:15.791420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:15.791404    2576 scope.go:117] "RemoveContainer" containerID="f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670"
Sep 04 22:34:15.791734 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:15.791711    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670\": container with ID starting with f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670 not found: ID does not exist" containerID="f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670"
Sep 04 22:34:15.791776 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:15.791750    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670"} err="failed to get container status \"f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670\": rpc error: code = NotFound desc = could not find container \"f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670\": container with ID starting with f1d02353339032de307165b9b3a0a25f2587879b22b60ec48e74ccf66eb7a670 not found: ID does not exist"
Sep 04 22:34:15.795507 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:15.795482    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7743-8f0b21\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="7603fc96-607b-4466-b9ae-057042a24ac3" pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:16.919163 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:16.919118    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7743-8f0b21\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="7603fc96-607b-4466-b9ae-057042a24ac3" pod="opendatahub/mt-curl-7743-8f0b21"
Sep 04 22:34:16.921240 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:16.921212    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7603fc96-607b-4466-b9ae-057042a24ac3" path="/var/lib/kubelet/pods/7603fc96-607b-4466-b9ae-057042a24ac3/volumes"
Sep 04 22:34:16.958102 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:16.958023    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:16.965473 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:16.965431    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7743-8dfaea\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:17.144075 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.143958    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8vkk\" (UniqueName: \"kubernetes.io/projected/be70e249-56c9-44ea-805e-6d3131bd3bfa-kube-api-access-t8vkk\") pod \"be70e249-56c9-44ea-805e-6d3131bd3bfa\" (UID: \"be70e249-56c9-44ea-805e-6d3131bd3bfa\") "
Sep 04 22:34:17.146137 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.146103    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be70e249-56c9-44ea-805e-6d3131bd3bfa-kube-api-access-t8vkk" (OuterVolumeSpecName: "kube-api-access-t8vkk") pod "be70e249-56c9-44ea-805e-6d3131bd3bfa" (UID: "be70e249-56c9-44ea-805e-6d3131bd3bfa"). InnerVolumeSpecName "kube-api-access-t8vkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:34:17.245771 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.245738    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8vkk\" (UniqueName: \"kubernetes.io/projected/be70e249-56c9-44ea-805e-6d3131bd3bfa-kube-api-access-t8vkk\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:17.286670 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.286626    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:17.348785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.348753    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"]
Sep 04 22:34:17.391137 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:17.391090    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7743-8dfaea\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:17.564231 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.564199    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:17.795489 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.795456    2576 generic.go:356] "Generic (PLEG): container finished" podID="be70e249-56c9-44ea-805e-6d3131bd3bfa" containerID="4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc" exitCode=137
Sep 04 22:34:17.795668 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.795496    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:17.795668 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.795567    2576 scope.go:117] "RemoveContainer" containerID="4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc"
Sep 04 22:34:17.798789 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:17.798759    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7743-8dfaea\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:17.804472 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.804455    2576 scope.go:117] "RemoveContainer" containerID="4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc"
Sep 04 22:34:17.804748 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:17.804730    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc\": container with ID starting with 4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc not found: ID does not exist" containerID="4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc"
Sep 04 22:34:17.804807 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:17.804756    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc"} err="failed to get container status \"4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc\": rpc error: code = NotFound desc = could not find container \"4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc\": container with ID starting with 4aff5563e9715f97ba39259c3cdf340a242be788f380d0bc75ad0a0f084fffcc not found: ID does not exist"
Sep 04 22:34:17.806168 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:17.806143    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7743-8dfaea\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" pod="opendatahub/mt-curl-7743-8dfaea"
Sep 04 22:34:18.919904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:18.919861    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" path="/var/lib/kubelet/pods/be70e249-56c9-44ea-805e-6d3131bd3bfa/volumes"
Sep 04 22:34:21.049037 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.048991    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:34:21.382244 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.382156    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"]
Sep 04 22:34:21.382442 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.382418    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh" podUID="edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" containerName="maas-api" containerID="cri-o://78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57" gracePeriod=30
Sep 04 22:34:21.407256 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.407221    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh" podUID="edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.67:8443/health\": dial tcp 10.133.0.67:8443: connect: connection refused"
Sep 04 22:34:21.498694 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.498149    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:21.645723 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.645665    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:34:21.691428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.691397    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-openshift-service-ca\") pod \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") "
Sep 04 22:34:21.691612 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.691450    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-metrics-certs\") pod \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") "
Sep 04 22:34:21.691612 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.691508    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-maas-api-tls\") pod \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") "
Sep 04 22:34:21.691612 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.691571    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skxcm\" (UniqueName: \"kubernetes.io/projected/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-kube-api-access-skxcm\") pod \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\" (UID: \"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4\") "
Sep 04 22:34:21.691790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.691768    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" (UID: "edf8bf1d-f21e-4fd6-bb64-0c70469eddf4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:34:21.691897 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.691876    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:21.693907 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.693835    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-kube-api-access-skxcm" (OuterVolumeSpecName: "kube-api-access-skxcm") pod "edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" (UID: "edf8bf1d-f21e-4fd6-bb64-0c70469eddf4"). InnerVolumeSpecName "kube-api-access-skxcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:34:21.694246 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.694209    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" (UID: "edf8bf1d-f21e-4fd6-bb64-0c70469eddf4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:34:21.694524 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.694499    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" (UID: "edf8bf1d-f21e-4fd6-bb64-0c70469eddf4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:34:21.710166 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.710108    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:21.792679 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.792632    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:21.792679 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.792670    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:21.792679 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.792684    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxcm\" (UniqueName: \"kubernetes.io/projected/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4-kube-api-access-skxcm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:21.818711 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.818662    2576 generic.go:356] "Generic (PLEG): container finished" podID="edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" containerID="78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57" exitCode=0
Sep 04 22:34:21.818888 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.818720    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"
Sep 04 22:34:21.818888 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.818719    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh" event={"ID":"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4","Type":"ContainerDied","Data":"78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57"}
Sep 04 22:34:21.818888 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.818792    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh" event={"ID":"edf8bf1d-f21e-4fd6-bb64-0c70469eddf4","Type":"ContainerDied","Data":"eceaac90c1e0dcbd0b469257beb77a11e7eb58c1403bd6571b1a0c7f0552fd47"}
Sep 04 22:34:21.818888 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.818825    2576 scope.go:117] "RemoveContainer" containerID="78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57"
Sep 04 22:34:21.828662 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.828639    2576 scope.go:117] "RemoveContainer" containerID="78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57"
Sep 04 22:34:21.828944 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:21.828922    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57\": container with ID starting with 78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57 not found: ID does not exist" containerID="78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57"
Sep 04 22:34:21.829016 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.828953    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57"} err="failed to get container status \"78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57\": rpc error: code = NotFound desc = could not find container \"78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57\": container with ID starting with 78fe8164465f95e0c9b8fce4ae38ff4df7d83906af2afc987ba860b6c158af57 not found: ID does not exist"
Sep 04 22:34:21.887200 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.887165    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"]
Sep 04 22:34:21.899029 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.898928    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b852ba-69c94db677-v6zsh"]
Sep 04 22:34:21.975414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:21.975358    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:22.110251 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:22.110207    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:22.472087 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:22.472034    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:22.918713 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:22.918614    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" path="/var/lib/kubelet/pods/edf8bf1d-f21e-4fd6-bb64-0c70469eddf4/volumes"
Sep 04 22:34:30.887400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:30.887336    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:31.047607 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:31.047566    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:31.228499 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:31.228460    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:32.239569 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.239506    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:32.369827 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.369788    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk" podUID="8f550c80-de80-4a5c-88b8-0d58542852c3" containerName="main" containerID="cri-o://9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16" gracePeriod=45
Sep 04 22:34:32.622267 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.622240    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:34:32.700299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700260    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f550c80-de80-4a5c-88b8-0d58542852c3-tls-certs\") pod \"8f550c80-de80-4a5c-88b8-0d58542852c3\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") "
Sep 04 22:34:32.700493 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700326    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-model-cache\") pod \"8f550c80-de80-4a5c-88b8-0d58542852c3\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") "
Sep 04 22:34:32.700493 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700433    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cplw9\" (UniqueName: \"kubernetes.io/projected/8f550c80-de80-4a5c-88b8-0d58542852c3-kube-api-access-cplw9\") pod \"8f550c80-de80-4a5c-88b8-0d58542852c3\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") "
Sep 04 22:34:32.700493 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700463    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-home\") pod \"8f550c80-de80-4a5c-88b8-0d58542852c3\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") "
Sep 04 22:34:32.700493 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700488    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-tmp-dir\") pod \"8f550c80-de80-4a5c-88b8-0d58542852c3\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") "
Sep 04 22:34:32.700698 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700548    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-dshm\") pod \"8f550c80-de80-4a5c-88b8-0d58542852c3\" (UID: \"8f550c80-de80-4a5c-88b8-0d58542852c3\") "
Sep 04 22:34:32.700698 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700649    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-model-cache" (OuterVolumeSpecName: "model-cache") pod "8f550c80-de80-4a5c-88b8-0d58542852c3" (UID: "8f550c80-de80-4a5c-88b8-0d58542852c3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:34:32.700795 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700736    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-home" (OuterVolumeSpecName: "home") pod "8f550c80-de80-4a5c-88b8-0d58542852c3" (UID: "8f550c80-de80-4a5c-88b8-0d58542852c3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:34:32.700857 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700815    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8f550c80-de80-4a5c-88b8-0d58542852c3" (UID: "8f550c80-de80-4a5c-88b8-0d58542852c3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:34:32.700987 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.700970    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:32.701081 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.701005    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:32.701081 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.701015    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:32.702517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.702489    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-dshm" (OuterVolumeSpecName: "dshm") pod "8f550c80-de80-4a5c-88b8-0d58542852c3" (UID: "8f550c80-de80-4a5c-88b8-0d58542852c3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:34:32.702622 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.702513    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f550c80-de80-4a5c-88b8-0d58542852c3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8f550c80-de80-4a5c-88b8-0d58542852c3" (UID: "8f550c80-de80-4a5c-88b8-0d58542852c3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:34:32.702622 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.702551    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f550c80-de80-4a5c-88b8-0d58542852c3-kube-api-access-cplw9" (OuterVolumeSpecName: "kube-api-access-cplw9") pod "8f550c80-de80-4a5c-88b8-0d58542852c3" (UID: "8f550c80-de80-4a5c-88b8-0d58542852c3"). InnerVolumeSpecName "kube-api-access-cplw9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:34:32.802417 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.802327    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8f550c80-de80-4a5c-88b8-0d58542852c3-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:32.802417 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.802359    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8f550c80-de80-4a5c-88b8-0d58542852c3-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:32.802417 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.802370    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cplw9\" (UniqueName: \"kubernetes.io/projected/8f550c80-de80-4a5c-88b8-0d58542852c3-kube-api-access-cplw9\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:34:32.887083 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.887029    2576 generic.go:356] "Generic (PLEG): container finished" podID="8f550c80-de80-4a5c-88b8-0d58542852c3" containerID="9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16" exitCode=0
Sep 04 22:34:32.887083 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.887088    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"
Sep 04 22:34:32.887309 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.887116    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk" event={"ID":"8f550c80-de80-4a5c-88b8-0d58542852c3","Type":"ContainerDied","Data":"9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16"}
Sep 04 22:34:32.887309 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.887162    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk" event={"ID":"8f550c80-de80-4a5c-88b8-0d58542852c3","Type":"ContainerDied","Data":"abe2fae305b72bbd3bb4685bb4808f6c057523e26d10be91fff07736644a79ad"}
Sep 04 22:34:32.887309 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.887178    2576 scope.go:117] "RemoveContainer" containerID="9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16"
Sep 04 22:34:32.896207 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.896182    2576 scope.go:117] "RemoveContainer" containerID="9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16"
Sep 04 22:34:32.896483 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:34:32.896463    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16\": container with ID starting with 9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16 not found: ID does not exist" containerID="9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16"
Sep 04 22:34:32.896531 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.896493    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16"} err="failed to get container status \"9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16\": rpc error: code = NotFound desc = could not find container \"9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16\": container with ID starting with 9684634bd163b17da37d852803fb270259a36ccd648603affb9145aeab556a16 not found: ID does not exist"
Sep 04 22:34:32.919883 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.919850    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"]
Sep 04 22:34:32.920530 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:32.920511    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/sub-test-model-bf86db-kserve-7fdc94d54c-2l2nk"]
Sep 04 22:34:34.919117 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:34.919085    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f550c80-de80-4a5c-88b8-0d58542852c3" path="/var/lib/kubelet/pods/8f550c80-de80-4a5c-88b8-0d58542852c3/volumes"
Sep 04 22:34:35.130946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:35.130908    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:34:36.859204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:36.859165    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:37.455741 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.455708    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"]
Sep 04 22:34:37.456202 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.456188    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" containerName="mt-curl-7743-8dfaea"
Sep 04 22:34:37.456252 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.456203    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" containerName="mt-curl-7743-8dfaea"
Sep 04 22:34:37.456252 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.456226    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" containerName="maas-api"
Sep 04 22:34:37.456252 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.456231    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" containerName="maas-api"
Sep 04 22:34:37.456252 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.456237    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f550c80-de80-4a5c-88b8-0d58542852c3" containerName="main"
Sep 04 22:34:37.456252 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.456243    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f550c80-de80-4a5c-88b8-0d58542852c3" containerName="main"
Sep 04 22:34:37.456409 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.456304    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="be70e249-56c9-44ea-805e-6d3131bd3bfa" containerName="mt-curl-7743-8dfaea"
Sep 04 22:34:37.456409 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.456315    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f550c80-de80-4a5c-88b8-0d58542852c3" containerName="main"
Sep 04 22:34:37.456409 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.456321    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf8bf1d-f21e-4fd6-bb64-0c70469eddf4" containerName="maas-api"
Sep 04 22:34:37.460978 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.460958    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.464350 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.464328    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bc62f0\"/\"default-dockercfg-7pmj4\""
Sep 04 22:34:37.464484 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.464328    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bc62f0\"/\"openshift-service-ca.crt\""
Sep 04 22:34:37.465209 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.465190    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-bc62f0\"/\"kube-root-ca.crt\""
Sep 04 22:34:37.465426 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.465405    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-bc62f0\"/\"rate-test-model-bc62f0-kserve-self-signed-certs\""
Sep 04 22:34:37.477218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.477188    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"]
Sep 04 22:34:37.508271 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.508227    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:37.550688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.550644    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-model-cache\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.550688 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.550692    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-tmp-dir\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.550880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.550754    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkl2w\" (UniqueName: \"kubernetes.io/projected/0b2a2583-7356-4dfa-9c12-f4110faceae0-kube-api-access-vkl2w\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.550880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.550837    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-dshm\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.550880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.550868    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-home\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.551002 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.550885    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b2a2583-7356-4dfa-9c12-f4110faceae0-tls-certs\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.651893 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.651854    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-model-cache\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.651893 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.651898    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-tmp-dir\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.652198 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.651918    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkl2w\" (UniqueName: \"kubernetes.io/projected/0b2a2583-7356-4dfa-9c12-f4110faceae0-kube-api-access-vkl2w\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.652198 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.651977    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-dshm\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.652198 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.652019    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-home\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.652198 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.652043    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b2a2583-7356-4dfa-9c12-f4110faceae0-tls-certs\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.652476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.652393    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-model-cache\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.652476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.652407    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-tmp-dir\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.652476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.652468    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-home\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.654442 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.654419    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-dshm\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.654546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.654528    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b2a2583-7356-4dfa-9c12-f4110faceae0-tls-certs\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.676404 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.676374    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkl2w\" (UniqueName: \"kubernetes.io/projected/0b2a2583-7356-4dfa-9c12-f4110faceae0-kube-api-access-vkl2w\") pod \"rate-test-model-bc62f0-kserve-6459467b55-2j9qf\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") " pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:37.770863 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:37.770768    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:38.064646 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:38.064617    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"]
Sep 04 22:34:38.067087 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:34:38.067043    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b2a2583_7356_4dfa_9c12_f4110faceae0.slice/crio-54223da8c3b3d7ab3ff1a435021db2b6ce4f67a121c8eb10854ff98470077729 WatchSource:0}: Error finding container 54223da8c3b3d7ab3ff1a435021db2b6ce4f67a121c8eb10854ff98470077729: Status 404 returned error can't find the container with id 54223da8c3b3d7ab3ff1a435021db2b6ce4f67a121c8eb10854ff98470077729
Sep 04 22:34:38.920977 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:38.920943    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf" event={"ID":"0b2a2583-7356-4dfa-9c12-f4110faceae0","Type":"ContainerStarted","Data":"73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89"}
Sep 04 22:34:38.920977 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:38.920978    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf" event={"ID":"0b2a2583-7356-4dfa-9c12-f4110faceae0","Type":"ContainerStarted","Data":"54223da8c3b3d7ab3ff1a435021db2b6ce4f67a121c8eb10854ff98470077729"}
Sep 04 22:34:38.949171 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:38.949114    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf" podStartSLOduration=1.949095363 podStartE2EDuration="1.949095363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:34:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:34:38.946572504 +0000 UTC m=+1562.465295037" watchObservedRunningTime="2026-09-04 22:34:38.949095363 +0000 UTC m=+1562.467817896"
Sep 04 22:34:41.965321 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:41.965279    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:46.164730 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:46.164686    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:47.771611 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:47.771572    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:47.772023 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:47.771629    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:47.784459 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:47.784426    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:47.980036 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:47.979998    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:34:48.290496 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:48.290456    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:48.451748 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:48.451701    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:48.576047 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:48.575963    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:48.695897 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:48.695852    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:48.808414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:48.808381    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:50.264628 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:50.264594    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:51.519656 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:51.519619    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:34:56.222657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:56.222620    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:56.342850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:56.342813    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:56.564880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:56.564767    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:34:57.083944 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:34:57.083906    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:00.721318 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.721282    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"]
Sep 04 22:35:00.725189 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.725165    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.729816 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.729794    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"kube-root-ca.crt\""
Sep 04 22:35:00.730645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.730625    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"openshift-service-ca.crt\""
Sep 04 22:35:00.730645 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.730640    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"rate-test-model-bf86db-kserve-self-signed-certs\""
Sep 04 22:35:00.730775 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.730627    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-bf86db\"/\"default-dockercfg-nzfvk\""
Sep 04 22:35:00.737311 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.737285    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"]
Sep 04 22:35:00.759590 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.759554    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:00.801673 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.801638    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-dshm\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.801673 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.801676    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-model-cache\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.801880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.801719    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432e94c3-3d71-42a2-985e-4db943954afa-tls-certs\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.801880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.801746    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzp5f\" (UniqueName: \"kubernetes.io/projected/432e94c3-3d71-42a2-985e-4db943954afa-kube-api-access-kzp5f\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.801880 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.801838    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-home\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.801986 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.801882    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-tmp-dir\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.903201 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.903145    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-dshm\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.903201 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.903204    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-model-cache\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.903510 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.903244    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432e94c3-3d71-42a2-985e-4db943954afa-tls-certs\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.903510 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.903280    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzp5f\" (UniqueName: \"kubernetes.io/projected/432e94c3-3d71-42a2-985e-4db943954afa-kube-api-access-kzp5f\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.903510 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.903352    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-home\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.903510 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.903396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-tmp-dir\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.903733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.903669    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-model-cache\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.903733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.903691    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-home\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.903823 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.903752    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-tmp-dir\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.905571 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.905549    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-dshm\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.905742 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.905726    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432e94c3-3d71-42a2-985e-4db943954afa-tls-certs\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:00.914871 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:00.914845    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzp5f\" (UniqueName: \"kubernetes.io/projected/432e94c3-3d71-42a2-985e-4db943954afa-kube-api-access-kzp5f\") pod \"rate-test-model-bf86db-kserve-6dc987bbbc-65njp\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") " pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:01.035731 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:01.035639    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:01.313263 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:01.313233    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"]
Sep 04 22:35:01.315230 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:35:01.315202    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod432e94c3_3d71_42a2_985e_4db943954afa.slice/crio-5d61c3a8e326fcd20b2745a4498c598c897403df992d6f5aaa47e7233b7f4047 WatchSource:0}: Error finding container 5d61c3a8e326fcd20b2745a4498c598c897403df992d6f5aaa47e7233b7f4047: Status 404 returned error can't find the container with id 5d61c3a8e326fcd20b2745a4498c598c897403df992d6f5aaa47e7233b7f4047
Sep 04 22:35:02.048514 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:02.048477    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp" event={"ID":"432e94c3-3d71-42a2-985e-4db943954afa","Type":"ContainerStarted","Data":"17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6"}
Sep 04 22:35:02.048514 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:02.048518    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp" event={"ID":"432e94c3-3d71-42a2-985e-4db943954afa","Type":"ContainerStarted","Data":"5d61c3a8e326fcd20b2745a4498c598c897403df992d6f5aaa47e7233b7f4047"}
Sep 04 22:35:02.072783 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:02.072727    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp" podStartSLOduration=2.072709419 podStartE2EDuration="2.072709419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:35:02.070642779 +0000 UTC m=+1585.589365313" watchObservedRunningTime="2026-09-04 22:35:02.072709419 +0000 UTC m=+1585.591431951"
Sep 04 22:35:02.177558 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:02.177523    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:04.754859 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:04.754810    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:35:06.434615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:06.434580    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:07.623201 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:07.623166    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq"]
Sep 04 22:35:07.627378 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:07.627345    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:07.632437 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:07.632413    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-f64b20-29809350-p28zq"]
Sep 04 22:35:08.918730 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:08.918699    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="715d23b7-fe71-4d06-bad4-1734489549d6" path="/var/lib/kubelet/pods/715d23b7-fe71-4d06-bad4-1734489549d6/volumes"
Sep 04 22:35:10.829206 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:10.829164    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:11.036039 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:11.035993    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:11.036039 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:11.036079    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:11.049068 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:11.049022    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:11.114517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:11.114433    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:17.235089 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:17.235035    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:17.481710 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:17.481665    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:17.813453 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:17.813410    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:18.535681 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:18.535638    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:20.929019 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:20.928977    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:35:29.031478 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.031390    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"]
Sep 04 22:35:29.031873 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.031634    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h" podUID="f71cbfce-2460-443b-82e3-9b041f946dcc" containerName="maas-api" containerID="cri-o://9568f32b34949808fcc5718bc029d7af84ff5b153916bf6f082a0f6a6edc19b6" gracePeriod=30
Sep 04 22:35:29.083175 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.083122    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:29.197467 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.197433    2576 generic.go:356] "Generic (PLEG): container finished" podID="f71cbfce-2460-443b-82e3-9b041f946dcc" containerID="9568f32b34949808fcc5718bc029d7af84ff5b153916bf6f082a0f6a6edc19b6" exitCode=0
Sep 04 22:35:29.197637 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.197501    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h" event={"ID":"f71cbfce-2460-443b-82e3-9b041f946dcc","Type":"ContainerDied","Data":"9568f32b34949808fcc5718bc029d7af84ff5b153916bf6f082a0f6a6edc19b6"}
Sep 04 22:35:29.218430 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.218396    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"]
Sep 04 22:35:29.218660 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.218632    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl" podUID="46def525-17dd-4ecf-9f3d-5e311b85c486" containerName="payload-pre-processing" containerID="cri-o://005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27" gracePeriod=30
Sep 04 22:35:29.346078 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.346038    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:35:29.384314 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.384267    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:29.395353 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.395328    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71cbfce-2460-443b-82e3-9b041f946dcc-openshift-service-ca\") pod \"f71cbfce-2460-443b-82e3-9b041f946dcc\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") "
Sep 04 22:35:29.395467 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.395448    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-maas-api-tls\") pod \"f71cbfce-2460-443b-82e3-9b041f946dcc\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") "
Sep 04 22:35:29.395526 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.395508    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktvxt\" (UniqueName: \"kubernetes.io/projected/f71cbfce-2460-443b-82e3-9b041f946dcc-kube-api-access-ktvxt\") pod \"f71cbfce-2460-443b-82e3-9b041f946dcc\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") "
Sep 04 22:35:29.395577 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.395541    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-metrics-certs\") pod \"f71cbfce-2460-443b-82e3-9b041f946dcc\" (UID: \"f71cbfce-2460-443b-82e3-9b041f946dcc\") "
Sep 04 22:35:29.395890 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.395720    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f71cbfce-2460-443b-82e3-9b041f946dcc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f71cbfce-2460-443b-82e3-9b041f946dcc" (UID: "f71cbfce-2460-443b-82e3-9b041f946dcc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:35:29.395890 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.395862    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f71cbfce-2460-443b-82e3-9b041f946dcc-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:29.402514 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.402475    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f71cbfce-2460-443b-82e3-9b041f946dcc" (UID: "f71cbfce-2460-443b-82e3-9b041f946dcc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:35:29.402638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.402482    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f71cbfce-2460-443b-82e3-9b041f946dcc-kube-api-access-ktvxt" (OuterVolumeSpecName: "kube-api-access-ktvxt") pod "f71cbfce-2460-443b-82e3-9b041f946dcc" (UID: "f71cbfce-2460-443b-82e3-9b041f946dcc"). InnerVolumeSpecName "kube-api-access-ktvxt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:35:29.402638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.402524    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f71cbfce-2460-443b-82e3-9b041f946dcc" (UID: "f71cbfce-2460-443b-82e3-9b041f946dcc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:35:29.475722 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.475691    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:35:29.496925 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.496900    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46def525-17dd-4ecf-9f3d-5e311b85c486-plugins-config-volume\") pod \"46def525-17dd-4ecf-9f3d-5e311b85c486\" (UID: \"46def525-17dd-4ecf-9f3d-5e311b85c486\") "
Sep 04 22:35:29.497089 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.496985    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fkgf\" (UniqueName: \"kubernetes.io/projected/46def525-17dd-4ecf-9f3d-5e311b85c486-kube-api-access-7fkgf\") pod \"46def525-17dd-4ecf-9f3d-5e311b85c486\" (UID: \"46def525-17dd-4ecf-9f3d-5e311b85c486\") "
Sep 04 22:35:29.497331 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.497311    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:29.497331 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.497320    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46def525-17dd-4ecf-9f3d-5e311b85c486-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "46def525-17dd-4ecf-9f3d-5e311b85c486" (UID: "46def525-17dd-4ecf-9f3d-5e311b85c486"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:35:29.497442 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.497334    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktvxt\" (UniqueName: \"kubernetes.io/projected/f71cbfce-2460-443b-82e3-9b041f946dcc-kube-api-access-ktvxt\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:29.497442 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.497349    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f71cbfce-2460-443b-82e3-9b041f946dcc-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:29.498999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.498977    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46def525-17dd-4ecf-9f3d-5e311b85c486-kube-api-access-7fkgf" (OuterVolumeSpecName: "kube-api-access-7fkgf") pod "46def525-17dd-4ecf-9f3d-5e311b85c486" (UID: "46def525-17dd-4ecf-9f3d-5e311b85c486"). InnerVolumeSpecName "kube-api-access-7fkgf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:35:29.531089 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.531032    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:29.598357 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.598274    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/46def525-17dd-4ecf-9f3d-5e311b85c486-plugins-config-volume\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:29.598357 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.598305    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fkgf\" (UniqueName: \"kubernetes.io/projected/46def525-17dd-4ecf-9f3d-5e311b85c486-kube-api-access-7fkgf\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:29.639982 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.639937    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:29.753573 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:29.753535    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:30.099494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.099459    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:30.204532 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.204496    2576 generic.go:356] "Generic (PLEG): container finished" podID="46def525-17dd-4ecf-9f3d-5e311b85c486" containerID="005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27" exitCode=0
Sep 04 22:35:30.204705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.204539    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"
Sep 04 22:35:30.204705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.204575    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl" event={"ID":"46def525-17dd-4ecf-9f3d-5e311b85c486","Type":"ContainerDied","Data":"005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27"}
Sep 04 22:35:30.204705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.204641    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl" event={"ID":"46def525-17dd-4ecf-9f3d-5e311b85c486","Type":"ContainerDied","Data":"07450ffc6d65b11f55d172049f2fba4530efcb559548d7ae9e55340abca8de39"}
Sep 04 22:35:30.204705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.204674    2576 scope.go:117] "RemoveContainer" containerID="005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27"
Sep 04 22:35:30.206160 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.206134    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h" event={"ID":"f71cbfce-2460-443b-82e3-9b041f946dcc","Type":"ContainerDied","Data":"3575b5b574e23641e9236e57d43b5116b2dbfbdffc8e70af2e7f756d5318b28a"}
Sep 04 22:35:30.206298 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.206187    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"
Sep 04 22:35:30.214182 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.214165    2576 scope.go:117] "RemoveContainer" containerID="005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27"
Sep 04 22:35:30.214468 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:35:30.214449    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27\": container with ID starting with 005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27 not found: ID does not exist" containerID="005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27"
Sep 04 22:35:30.214537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.214477    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27"} err="failed to get container status \"005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27\": rpc error: code = NotFound desc = could not find container \"005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27\": container with ID starting with 005a432e4d17f85702b773566e794c46a830d02d919aa8a07393a412594f5b27 not found: ID does not exist"
Sep 04 22:35:30.214537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.214494    2576 scope.go:117] "RemoveContainer" containerID="9568f32b34949808fcc5718bc029d7af84ff5b153916bf6f082a0f6a6edc19b6"
Sep 04 22:35:30.245179 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.245154    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"]
Sep 04 22:35:30.254494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.254471    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-a12dcf-fb8866676-scv6h"]
Sep 04 22:35:30.279416 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.279381    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"]
Sep 04 22:35:30.291544 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.291518    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a12dcf-74bc76b98b-x9pjl"]
Sep 04 22:35:30.918160 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.918126    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46def525-17dd-4ecf-9f3d-5e311b85c486" path="/var/lib/kubelet/pods/46def525-17dd-4ecf-9f3d-5e311b85c486/volumes"
Sep 04 22:35:30.918655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:30.918635    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f71cbfce-2460-443b-82e3-9b041f946dcc" path="/var/lib/kubelet/pods/f71cbfce-2460-443b-82e3-9b041f946dcc/volumes"
Sep 04 22:35:37.006983 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:37.006944    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:37.375689 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:37.375599    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"]
Sep 04 22:35:37.375844 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:37.375804    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy" containerID="cri-o://6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5" gracePeriod=30
Sep 04 22:35:37.455200 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:37.455159    2576 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 22:35:37.455400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:37.455237    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:35:38.547082 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:38.547022    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:38.693584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:38.693548    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:35:38.779589 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:38.779536    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:42.635684 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.635651    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:35:42.646544 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.646475    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"]
Sep 04 22:35:42.711164 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.711131    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:42.730885 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.730855    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-certs\") pod \"0022b660-5109-4ef0-be95-c5805b6d0aef\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") "
Sep 04 22:35:42.731088 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.730895    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-socket\") pod \"0022b660-5109-4ef0-be95-c5805b6d0aef\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") "
Sep 04 22:35:42.731088 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731009    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-data\") pod \"0022b660-5109-4ef0-be95-c5805b6d0aef\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") "
Sep 04 22:35:42.731088 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731042    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68c49\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-kube-api-access-68c49\") pod \"0022b660-5109-4ef0-be95-c5805b6d0aef\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") "
Sep 04 22:35:42.731262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731087    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-credential-socket\") pod \"0022b660-5109-4ef0-be95-c5805b6d0aef\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") "
Sep 04 22:35:42.731262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731139    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0022b660-5109-4ef0-be95-c5805b6d0aef-istiod-ca-cert\") pod \"0022b660-5109-4ef0-be95-c5805b6d0aef\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") "
Sep 04 22:35:42.731262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731173    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-envoy\") pod \"0022b660-5109-4ef0-be95-c5805b6d0aef\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") "
Sep 04 22:35:42.731262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731212    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0022b660-5109-4ef0-be95-c5805b6d0aef" (UID: "0022b660-5109-4ef0-be95-c5805b6d0aef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:42.731262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731222    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-token\") pod \"0022b660-5109-4ef0-be95-c5805b6d0aef\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") "
Sep 04 22:35:42.731262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731253    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-podinfo\") pod \"0022b660-5109-4ef0-be95-c5805b6d0aef\" (UID: \"0022b660-5109-4ef0-be95-c5805b6d0aef\") "
Sep 04 22:35:42.731570 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731451    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0022b660-5109-4ef0-be95-c5805b6d0aef" (UID: "0022b660-5109-4ef0-be95-c5805b6d0aef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:42.731570 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731460    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0022b660-5109-4ef0-be95-c5805b6d0aef" (UID: "0022b660-5109-4ef0-be95-c5805b6d0aef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:42.731684 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731615    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-data" (OuterVolumeSpecName: "istio-data") pod "0022b660-5109-4ef0-be95-c5805b6d0aef" (UID: "0022b660-5109-4ef0-be95-c5805b6d0aef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:42.731684 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731659    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-credential-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:42.731684 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731680    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:42.731810 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731694    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-workload-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:42.731810 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.731703    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0022b660-5109-4ef0-be95-c5805b6d0aef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0022b660-5109-4ef0-be95-c5805b6d0aef" (UID: "0022b660-5109-4ef0-be95-c5805b6d0aef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:35:42.733634 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.733611    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0022b660-5109-4ef0-be95-c5805b6d0aef" (UID: "0022b660-5109-4ef0-be95-c5805b6d0aef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:42.733789 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.733689    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-kube-api-access-68c49" (OuterVolumeSpecName: "kube-api-access-68c49") pod "0022b660-5109-4ef0-be95-c5805b6d0aef" (UID: "0022b660-5109-4ef0-be95-c5805b6d0aef"). InnerVolumeSpecName "kube-api-access-68c49". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:35:42.734034 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.733949    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0022b660-5109-4ef0-be95-c5805b6d0aef" (UID: "0022b660-5109-4ef0-be95-c5805b6d0aef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 22:35:42.734034 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.733992    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-token" (OuterVolumeSpecName: "istio-token") pod "0022b660-5109-4ef0-be95-c5805b6d0aef" (UID: "0022b660-5109-4ef0-be95-c5805b6d0aef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:35:42.833218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.833179    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-data\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:42.833218 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.833216    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68c49\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-kube-api-access-68c49\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:42.833423 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.833232    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0022b660-5109-4ef0-be95-c5805b6d0aef-istiod-ca-cert\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:42.833423 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.833245    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-envoy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:42.833423 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.833257    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-token\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:42.833423 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.833268    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0022b660-5109-4ef0-be95-c5805b6d0aef-istio-podinfo\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:42.852856 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:42.852820    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:43.293523 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.293485    2576 generic.go:356] "Generic (PLEG): container finished" podID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerID="6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5" exitCode=0
Sep 04 22:35:43.293704 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.293540    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"
Sep 04 22:35:43.293704 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.293563    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" event={"ID":"0022b660-5109-4ef0-be95-c5805b6d0aef","Type":"ContainerDied","Data":"6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5"}
Sep 04 22:35:43.293704 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.293641    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7" event={"ID":"0022b660-5109-4ef0-be95-c5805b6d0aef","Type":"ContainerDied","Data":"35dc5770fdb14cb540f9f2e6f5857ebd46e9b4d8bc1c72e9b8d93dfc9f0ea94b"}
Sep 04 22:35:43.293704 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.293664    2576 scope.go:117] "RemoveContainer" containerID="6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5"
Sep 04 22:35:43.304156 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.304135    2576 scope.go:117] "RemoveContainer" containerID="6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1"
Sep 04 22:35:43.312238 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.312219    2576 scope.go:117] "RemoveContainer" containerID="6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5"
Sep 04 22:35:43.312536 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:35:43.312515    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5\": container with ID starting with 6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5 not found: ID does not exist" containerID="6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5"
Sep 04 22:35:43.312629 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.312546    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5"} err="failed to get container status \"6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5\": rpc error: code = NotFound desc = could not find container \"6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5\": container with ID starting with 6de7d31defcc7695724d88da7c415db07fb9b1a72c5b6398774b695416cdbcf5 not found: ID does not exist"
Sep 04 22:35:43.312629 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.312570    2576 scope.go:117] "RemoveContainer" containerID="6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1"
Sep 04 22:35:43.312865 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:35:43.312847    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1\": container with ID starting with 6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1 not found: ID does not exist" containerID="6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1"
Sep 04 22:35:43.312908 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.312873    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1"} err="failed to get container status \"6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1\": rpc error: code = NotFound desc = could not find container \"6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1\": container with ID starting with 6b3833cfdae4b5c1999533dc79f72ad6545d16a827630493cf818270731be5b1 not found: ID does not exist"
Sep 04 22:35:43.328803 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.328779    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"]
Sep 04 22:35:43.333512 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.333482    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a12dcf-openshift-default-67947f977d-ghwm7"]
Sep 04 22:35:43.619091 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.618984    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"]
Sep 04 22:35:43.732821 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:43.732788    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:44.184870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:44.184834    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:44.817885 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:44.817849    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:44.919651 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:44.919617    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" path="/var/lib/kubelet/pods/0022b660-5109-4ef0-be95-c5805b6d0aef/volumes"
Sep 04 22:35:44.984703 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:44.984667    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:48.497743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:48.497706    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:48.779208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:48.779110    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:49.588024 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.587986    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"]
Sep 04 22:35:49.588455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588398    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46def525-17dd-4ecf-9f3d-5e311b85c486" containerName="payload-pre-processing"
Sep 04 22:35:49.588455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588412    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="46def525-17dd-4ecf-9f3d-5e311b85c486" containerName="payload-pre-processing"
Sep 04 22:35:49.588455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588428    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f71cbfce-2460-443b-82e3-9b041f946dcc" containerName="maas-api"
Sep 04 22:35:49.588455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588434    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f71cbfce-2460-443b-82e3-9b041f946dcc" containerName="maas-api"
Sep 04 22:35:49.588455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588455    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy"
Sep 04 22:35:49.588618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588461    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy"
Sep 04 22:35:49.588618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588467    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy"
Sep 04 22:35:49.588618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588472    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy"
Sep 04 22:35:49.588618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588539    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy"
Sep 04 22:35:49.588618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588548    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="46def525-17dd-4ecf-9f3d-5e311b85c486" containerName="payload-pre-processing"
Sep 04 22:35:49.588618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588556    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0022b660-5109-4ef0-be95-c5805b6d0aef" containerName="istio-proxy"
Sep 04 22:35:49.588618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.588563    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f71cbfce-2460-443b-82e3-9b041f946dcc" containerName="maas-api"
Sep 04 22:35:49.593036 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.593020    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.595488 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:35:49.595459    2576 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" podUID="68c88a21-4797-4c81-b408-b84b3f359957" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.595733 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:35:49.595706    2576 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"maas-api-e2e-ipp-a-2df128-serving-cert\" is forbidden: User \"system:node:ip-10-0-139-78.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-78.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2df128-serving-cert\"" type="*v1.Secret"
Sep 04 22:35:49.631177 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.631146    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"]
Sep 04 22:35:49.703873 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.703835    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-metrics-certs\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.704093 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.703889    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsdcn\" (UniqueName: \"kubernetes.io/projected/68c88a21-4797-4c81-b408-b84b3f359957-kube-api-access-tsdcn\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.704093 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.704009    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c88a21-4797-4c81-b408-b84b3f359957-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.704212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.704119    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.731267 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.731233    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:49.805492 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.805457    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsdcn\" (UniqueName: \"kubernetes.io/projected/68c88a21-4797-4c81-b408-b84b3f359957-kube-api-access-tsdcn\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.805707 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.805521    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c88a21-4797-4c81-b408-b84b3f359957-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.805707 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.805559    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.805707 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.805653    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-metrics-certs\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.806303 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.806279    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c88a21-4797-4c81-b408-b84b3f359957-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.808066 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.808034    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-metrics-certs\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:49.831829 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:49.831799    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsdcn\" (UniqueName: \"kubernetes.io/projected/68c88a21-4797-4c81-b408-b84b3f359957-kube-api-access-tsdcn\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:50.033578 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:50.033544    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:50.738608 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:50.738573    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-2df128-serving-cert\""
Sep 04 22:35:50.748318 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:50.748286    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-maas-api-tls\") pod \"maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:50.804197 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:50.804161    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:50.830695 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:50.830660    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:50.993854 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:50.993760    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:35:51.109901 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:51.109871    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"]
Sep 04 22:35:51.117065 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:35:51.117012    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c88a21_4797_4c81_b408_b84b3f359957.slice/crio-3dc61736d25819d46f93ed8259981733425d74349746b2ff6c0a3aa7edd8ee1b WatchSource:0}: Error finding container 3dc61736d25819d46f93ed8259981733425d74349746b2ff6c0a3aa7edd8ee1b: Status 404 returned error can't find the container with id 3dc61736d25819d46f93ed8259981733425d74349746b2ff6c0a3aa7edd8ee1b
Sep 04 22:35:51.338541 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:51.338456    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm" event={"ID":"68c88a21-4797-4c81-b408-b84b3f359957","Type":"ContainerStarted","Data":"3dc61736d25819d46f93ed8259981733425d74349746b2ff6c0a3aa7edd8ee1b"}
Sep 04 22:35:52.346258 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:52.346223    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm" event={"ID":"68c88a21-4797-4c81-b408-b84b3f359957","Type":"ContainerStarted","Data":"2671bc997c1333376e1c3d2d4241ff7eb74ac64363789a05e1316b3b0efa8539"}
Sep 04 22:35:52.346722 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:52.346306    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:52.397646 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:52.397584    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm" podStartSLOduration=2.921637065 podStartE2EDuration="3.397566461s" totalImagesPullingTime="475.929396ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:35:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:35:52.396179323 +0000 UTC m=+1635.914901931" watchObservedRunningTime="2026-09-04 22:35:52.397566461 +0000 UTC m=+1635.916288994"
Sep 04 22:35:53.515103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:53.515041    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:35:57.665816 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:57.665775    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf" podUID="0b2a2583-7356-4dfa-9c12-f4110faceae0" containerName="main" containerID="cri-o://73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89" gracePeriod=45
Sep 04 22:35:57.927849 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:57.927783    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:35:58.087963 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.087920    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-dshm\") pod \"0b2a2583-7356-4dfa-9c12-f4110faceae0\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") "
Sep 04 22:35:58.088155 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088020    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b2a2583-7356-4dfa-9c12-f4110faceae0-tls-certs\") pod \"0b2a2583-7356-4dfa-9c12-f4110faceae0\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") "
Sep 04 22:35:58.088155 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088050    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vkl2w\" (UniqueName: \"kubernetes.io/projected/0b2a2583-7356-4dfa-9c12-f4110faceae0-kube-api-access-vkl2w\") pod \"0b2a2583-7356-4dfa-9c12-f4110faceae0\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") "
Sep 04 22:35:58.088155 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088132    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-tmp-dir\") pod \"0b2a2583-7356-4dfa-9c12-f4110faceae0\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") "
Sep 04 22:35:58.088309 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088284    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-home\") pod \"0b2a2583-7356-4dfa-9c12-f4110faceae0\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") "
Sep 04 22:35:58.088357 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088340    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-model-cache\") pod \"0b2a2583-7356-4dfa-9c12-f4110faceae0\" (UID: \"0b2a2583-7356-4dfa-9c12-f4110faceae0\") "
Sep 04 22:35:58.088414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088387    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0b2a2583-7356-4dfa-9c12-f4110faceae0" (UID: "0b2a2583-7356-4dfa-9c12-f4110faceae0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:58.088564 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088523    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-home" (OuterVolumeSpecName: "home") pod "0b2a2583-7356-4dfa-9c12-f4110faceae0" (UID: "0b2a2583-7356-4dfa-9c12-f4110faceae0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:58.088712 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088680    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-model-cache" (OuterVolumeSpecName: "model-cache") pod "0b2a2583-7356-4dfa-9c12-f4110faceae0" (UID: "0b2a2583-7356-4dfa-9c12-f4110faceae0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:58.089212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088855    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:58.089212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088879    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:58.089212 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.088893    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:58.090361 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.090337    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-dshm" (OuterVolumeSpecName: "dshm") pod "0b2a2583-7356-4dfa-9c12-f4110faceae0" (UID: "0b2a2583-7356-4dfa-9c12-f4110faceae0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:58.090453 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.090386    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b2a2583-7356-4dfa-9c12-f4110faceae0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0b2a2583-7356-4dfa-9c12-f4110faceae0" (UID: "0b2a2583-7356-4dfa-9c12-f4110faceae0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:35:58.090453 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.090392    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b2a2583-7356-4dfa-9c12-f4110faceae0-kube-api-access-vkl2w" (OuterVolumeSpecName: "kube-api-access-vkl2w") pod "0b2a2583-7356-4dfa-9c12-f4110faceae0" (UID: "0b2a2583-7356-4dfa-9c12-f4110faceae0"). InnerVolumeSpecName "kube-api-access-vkl2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:35:58.190424 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.190337    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0b2a2583-7356-4dfa-9c12-f4110faceae0-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:58.190424 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.190370    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0b2a2583-7356-4dfa-9c12-f4110faceae0-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:58.190424 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.190381    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkl2w\" (UniqueName: \"kubernetes.io/projected/0b2a2583-7356-4dfa-9c12-f4110faceae0-kube-api-access-vkl2w\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:58.357366 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.357336    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:35:58.378314 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.378279    2576 generic.go:356] "Generic (PLEG): container finished" podID="0b2a2583-7356-4dfa-9c12-f4110faceae0" containerID="73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89" exitCode=0
Sep 04 22:35:58.378471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.378321    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"
Sep 04 22:35:58.378471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.378365    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf" event={"ID":"0b2a2583-7356-4dfa-9c12-f4110faceae0","Type":"ContainerDied","Data":"73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89"}
Sep 04 22:35:58.378471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.378407    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf" event={"ID":"0b2a2583-7356-4dfa-9c12-f4110faceae0","Type":"ContainerDied","Data":"54223da8c3b3d7ab3ff1a435021db2b6ce4f67a121c8eb10854ff98470077729"}
Sep 04 22:35:58.378471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.378425    2576 scope.go:117] "RemoveContainer" containerID="73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89"
Sep 04 22:35:58.388642 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.388618    2576 scope.go:117] "RemoveContainer" containerID="73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89"
Sep 04 22:35:58.388943 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:35:58.388923    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89\": container with ID starting with 73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89 not found: ID does not exist" containerID="73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89"
Sep 04 22:35:58.389006 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.388951    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89"} err="failed to get container status \"73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89\": rpc error: code = NotFound desc = could not find container \"73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89\": container with ID starting with 73fcf04fc70cb26ee96bcb7b9b07709cbb668cc5405d858fa07570375ad90f89 not found: ID does not exist"
Sep 04 22:35:58.419174 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.419139    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"]
Sep 04 22:35:58.423381 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.423347    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-bc62f0/rate-test-model-bc62f0-kserve-6459467b55-2j9qf"]
Sep 04 22:35:58.638877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.638840    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp" podUID="432e94c3-3d71-42a2-985e-4db943954afa" containerName="main" containerID="cri-o://17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6" gracePeriod=45
Sep 04 22:35:58.897448 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.897386    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:58.919670 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.919634    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b2a2583-7356-4dfa-9c12-f4110faceae0" path="/var/lib/kubelet/pods/0b2a2583-7356-4dfa-9c12-f4110faceae0/volumes"
Sep 04 22:35:58.997015 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.996978    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-home\") pod \"432e94c3-3d71-42a2-985e-4db943954afa\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") "
Sep 04 22:35:58.997015 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997016    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzp5f\" (UniqueName: \"kubernetes.io/projected/432e94c3-3d71-42a2-985e-4db943954afa-kube-api-access-kzp5f\") pod \"432e94c3-3d71-42a2-985e-4db943954afa\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") "
Sep 04 22:35:58.997279 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997088    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-model-cache\") pod \"432e94c3-3d71-42a2-985e-4db943954afa\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") "
Sep 04 22:35:58.997279 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997107    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432e94c3-3d71-42a2-985e-4db943954afa-tls-certs\") pod \"432e94c3-3d71-42a2-985e-4db943954afa\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") "
Sep 04 22:35:58.997279 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997130    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-dshm\") pod \"432e94c3-3d71-42a2-985e-4db943954afa\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") "
Sep 04 22:35:58.997279 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997180    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-tmp-dir\") pod \"432e94c3-3d71-42a2-985e-4db943954afa\" (UID: \"432e94c3-3d71-42a2-985e-4db943954afa\") "
Sep 04 22:35:58.997495 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997329    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-home" (OuterVolumeSpecName: "home") pod "432e94c3-3d71-42a2-985e-4db943954afa" (UID: "432e94c3-3d71-42a2-985e-4db943954afa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:58.997495 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997386    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-model-cache" (OuterVolumeSpecName: "model-cache") pod "432e94c3-3d71-42a2-985e-4db943954afa" (UID: "432e94c3-3d71-42a2-985e-4db943954afa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:58.997592 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997509    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "432e94c3-3d71-42a2-985e-4db943954afa" (UID: "432e94c3-3d71-42a2-985e-4db943954afa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:58.997630 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997589    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:58.997630 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997607    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:58.997630 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.997617    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:58.999290 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.999267    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-dshm" (OuterVolumeSpecName: "dshm") pod "432e94c3-3d71-42a2-985e-4db943954afa" (UID: "432e94c3-3d71-42a2-985e-4db943954afa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:35:58.999391 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.999298    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/432e94c3-3d71-42a2-985e-4db943954afa-kube-api-access-kzp5f" (OuterVolumeSpecName: "kube-api-access-kzp5f") pod "432e94c3-3d71-42a2-985e-4db943954afa" (UID: "432e94c3-3d71-42a2-985e-4db943954afa"). InnerVolumeSpecName "kube-api-access-kzp5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:35:58.999391 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:58.999362    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/432e94c3-3d71-42a2-985e-4db943954afa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "432e94c3-3d71-42a2-985e-4db943954afa" (UID: "432e94c3-3d71-42a2-985e-4db943954afa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:35:59.099174 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.099131    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzp5f\" (UniqueName: \"kubernetes.io/projected/432e94c3-3d71-42a2-985e-4db943954afa-kube-api-access-kzp5f\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:59.099174 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.099168    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/432e94c3-3d71-42a2-985e-4db943954afa-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:59.099174 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.099181    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/432e94c3-3d71-42a2-985e-4db943954afa-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:35:59.384545 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.384511    2576 generic.go:356] "Generic (PLEG): container finished" podID="432e94c3-3d71-42a2-985e-4db943954afa" containerID="17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6" exitCode=0
Sep 04 22:35:59.384545 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.384547    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"
Sep 04 22:35:59.384817 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.384596    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp" event={"ID":"432e94c3-3d71-42a2-985e-4db943954afa","Type":"ContainerDied","Data":"17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6"}
Sep 04 22:35:59.384817 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.384638    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp" event={"ID":"432e94c3-3d71-42a2-985e-4db943954afa","Type":"ContainerDied","Data":"5d61c3a8e326fcd20b2745a4498c598c897403df992d6f5aaa47e7233b7f4047"}
Sep 04 22:35:59.384817 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.384655    2576 scope.go:117] "RemoveContainer" containerID="17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6"
Sep 04 22:35:59.394491 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.394471    2576 scope.go:117] "RemoveContainer" containerID="17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6"
Sep 04 22:35:59.394772 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:35:59.394754    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6\": container with ID starting with 17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6 not found: ID does not exist" containerID="17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6"
Sep 04 22:35:59.394821 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.394780    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6"} err="failed to get container status \"17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6\": rpc error: code = NotFound desc = could not find container \"17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6\": container with ID starting with 17f0d2c0b4022f0b6739e793859eb62a54684dd3c6c6f4d87daeaa7bef6fa5b6 not found: ID does not exist"
Sep 04 22:35:59.410850 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.410820    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"]
Sep 04 22:35:59.415455 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:35:59.415431    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-bf86db/rate-test-model-bf86db-kserve-6dc987bbbc-65njp"]
Sep 04 22:36:00.918699 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:00.918659    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="432e94c3-3d71-42a2-985e-4db943954afa" path="/var/lib/kubelet/pods/432e94c3-3d71-42a2-985e-4db943954afa/volumes"
Sep 04 22:36:13.249862 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:13.249822    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:14.217691 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.217649    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"]
Sep 04 22:36:14.218367 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.218334    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="432e94c3-3d71-42a2-985e-4db943954afa" containerName="main"
Sep 04 22:36:14.218367 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.218364    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="432e94c3-3d71-42a2-985e-4db943954afa" containerName="main"
Sep 04 22:36:14.218546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.218408    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b2a2583-7356-4dfa-9c12-f4110faceae0" containerName="main"
Sep 04 22:36:14.218546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.218418    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b2a2583-7356-4dfa-9c12-f4110faceae0" containerName="main"
Sep 04 22:36:14.218546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.218515    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b2a2583-7356-4dfa-9c12-f4110faceae0" containerName="main"
Sep 04 22:36:14.218546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.218534    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="432e94c3-3d71-42a2-985e-4db943954afa" containerName="main"
Sep 04 22:36:14.222139 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.222116    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.225410 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.225385    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-ecfcad-openshift-default-dockercfg-gbc7b\""
Sep 04 22:36:14.236010 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.235983    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"]
Sep 04 22:36:14.352121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.352077    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001de2e3-d82e-4072-94f3-da88c216f903-istiod-ca-cert\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.352121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.352127    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-istio-token\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.352596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.352182    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001de2e3-d82e-4072-94f3-da88c216f903-istio-podinfo\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.352596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.352215    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-certs\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.352596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.352238    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-socket\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.352596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.352290    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-credential-socket\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.352596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.352308    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvbpv\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-kube-api-access-bvbpv\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.352596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.352326    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-data\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.352596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.352393    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-envoy\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.422945 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.422911    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:14.452964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.452929    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001de2e3-d82e-4072-94f3-da88c216f903-istiod-ca-cert\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.452975    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-istio-token\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453023    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001de2e3-d82e-4072-94f3-da88c216f903-istio-podinfo\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-certs\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453331 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453266    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-socket\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453390 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453335    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-credential-socket\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453390 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453369    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvbpv\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-kube-api-access-bvbpv\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453396    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-data\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453435    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-envoy\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453498 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453467    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-certs\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453669 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453635    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-credential-socket\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453717 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453663    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-socket\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453769 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453732    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001de2e3-d82e-4072-94f3-da88c216f903-istiod-ca-cert\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.453821 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.453803    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-data\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.455567 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.455547    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-envoy\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.455717 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.455696    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001de2e3-d82e-4072-94f3-da88c216f903-istio-podinfo\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.467733 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.467665    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvbpv\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-kube-api-access-bvbpv\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.468210 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.468192    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-istio-token\") pod \"e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") " pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.520148 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.520109    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:14.536451 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.536417    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:14.817940 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.817733    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"]
Sep 04 22:36:14.820781 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:36:14.820750    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod001de2e3_d82e_4072_94f3_da88c216f903.slice/crio-61ab714e926c164cda90021f91d3ffd7a8e4632f453c9e2eea135816d77f1b64 WatchSource:0}: Error finding container 61ab714e926c164cda90021f91d3ffd7a8e4632f453c9e2eea135816d77f1b64: Status 404 returned error can't find the container with id 61ab714e926c164cda90021f91d3ffd7a8e4632f453c9e2eea135816d77f1b64
Sep 04 22:36:14.822955 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.822923    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:36:14.823018 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.822991    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:36:14.823084 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:14.823020    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:36:15.473695 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:15.473655    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" event={"ID":"001de2e3-d82e-4072-94f3-da88c216f903","Type":"ContainerStarted","Data":"b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef"}
Sep 04 22:36:15.474094 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:15.473705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" event={"ID":"001de2e3-d82e-4072-94f3-da88c216f903","Type":"ContainerStarted","Data":"61ab714e926c164cda90021f91d3ffd7a8e4632f453c9e2eea135816d77f1b64"}
Sep 04 22:36:15.503465 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:15.503401    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" podStartSLOduration=1.50338351 podStartE2EDuration="1.50338351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:36:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:36:15.501520426 +0000 UTC m=+1659.020242959" watchObservedRunningTime="2026-09-04 22:36:15.50338351 +0000 UTC m=+1659.022106043"
Sep 04 22:36:15.537451 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:15.537414    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:15.542583 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:15.542559    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:16.478933 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:16.478901    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:16.480101 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:16.480079    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:17.544065 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:17.544016    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:17.663368 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:17.663332    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:18.773414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:18.773374    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:19.663554 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.663522    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"]
Sep 04 22:36:19.666746 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.666730    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:19.670314 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.670290    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ecfcad-dockercfg-tqkwj\""
Sep 04 22:36:19.670735 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.670717    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ecfcad\""
Sep 04 22:36:19.683430 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.683406    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"]
Sep 04 22:36:19.807416 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.807380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea988877-6d14-4178-b514-982b54801708-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5\" (UID: \"ea988877-6d14-4178-b514-982b54801708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:19.807845 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.807452    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhvcg\" (UniqueName: \"kubernetes.io/projected/ea988877-6d14-4178-b514-982b54801708-kube-api-access-lhvcg\") pod \"payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5\" (UID: \"ea988877-6d14-4178-b514-982b54801708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:19.908343 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.908303    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea988877-6d14-4178-b514-982b54801708-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5\" (UID: \"ea988877-6d14-4178-b514-982b54801708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:19.908513 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.908355    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvcg\" (UniqueName: \"kubernetes.io/projected/ea988877-6d14-4178-b514-982b54801708-kube-api-access-lhvcg\") pod \"payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5\" (UID: \"ea988877-6d14-4178-b514-982b54801708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:19.908977 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.908956    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea988877-6d14-4178-b514-982b54801708-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5\" (UID: \"ea988877-6d14-4178-b514-982b54801708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:19.930580 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.930523    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhvcg\" (UniqueName: \"kubernetes.io/projected/ea988877-6d14-4178-b514-982b54801708-kube-api-access-lhvcg\") pod \"payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5\" (UID: \"ea988877-6d14-4178-b514-982b54801708\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:19.977160 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:19.977121    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:20.255271 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:20.255244    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"]
Sep 04 22:36:20.257961 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:36:20.257935    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea988877_6d14_4178_b514_982b54801708.slice/crio-4b3abf2e950857d81f1b5f671e12d273b46e36f24d3646a4b85cd6c499e73ce4 WatchSource:0}: Error finding container 4b3abf2e950857d81f1b5f671e12d273b46e36f24d3646a4b85cd6c499e73ce4: Status 404 returned error can't find the container with id 4b3abf2e950857d81f1b5f671e12d273b46e36f24d3646a4b85cd6c499e73ce4
Sep 04 22:36:20.278418 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:20.278385    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:20.432858 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:20.432823    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:20.502424 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:20.502389    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5" event={"ID":"ea988877-6d14-4178-b514-982b54801708","Type":"ContainerStarted","Data":"b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d"}
Sep 04 22:36:20.502632 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:20.502471    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5" event={"ID":"ea988877-6d14-4178-b514-982b54801708","Type":"ContainerStarted","Data":"4b3abf2e950857d81f1b5f671e12d273b46e36f24d3646a4b85cd6c499e73ce4"}
Sep 04 22:36:20.502632 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:20.502507    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:20.525241 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:20.525145    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5" podStartSLOduration=1.5251298869999999 podStartE2EDuration="1.525129887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:36:20.52367427 +0000 UTC m=+1664.042396805" watchObservedRunningTime="2026-09-04 22:36:20.525129887 +0000 UTC m=+1664.043852420"
Sep 04 22:36:24.720541 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:24.720503    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:36:31.510103 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:31.510048    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:36:35.588118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:35.588083    2576 generic.go:356] "Generic (PLEG): container finished" podID="001de2e3-d82e-4072-94f3-da88c216f903" containerID="b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef" exitCode=0
Sep 04 22:36:35.588555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:35.588158    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" event={"ID":"001de2e3-d82e-4072-94f3-da88c216f903","Type":"ContainerDied","Data":"b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef"}
Sep 04 22:36:35.588604 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:35.588565    2576 scope.go:117] "RemoveContainer" containerID="b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef"
Sep 04 22:36:35.589656 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:35.589624    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:36:35.589768 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:35.589713    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:36:35.589768 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:35.589762    2576 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 04 22:36:36.596677 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:36.596640    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" event={"ID":"001de2e3-d82e-4072-94f3-da88c216f903","Type":"ContainerStarted","Data":"8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11"}
Sep 04 22:36:37.069553 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:37.069502    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:37.180964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:37.180924    2576 scope.go:117] "RemoveContainer" containerID="aecb0d69cef567e689fc521c4e9be203fa533ede68b4a7d84befb312b95ac223"
Sep 04 22:36:37.537311 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:37.537276    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:37.537492 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:37.537465    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 04 22:36:37.537542 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:37.537511    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 04 22:36:38.537223 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:38.537184    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 04 22:36:38.537634 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:38.537241    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 04 22:36:39.536803 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:39.536762    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 04 22:36:39.537020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:39.536823    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 04 22:36:40.537332 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:40.537296    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 04 22:36:40.537743 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:40.537350    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 04 22:36:41.537476 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:41.537431    2576 patch_prober.go:28] interesting pod/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Sep 04 22:36:41.537887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:41.537509    2576 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Sep 04 22:36:42.496906 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:42.496871    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:42.541132 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:42.541101    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:42.541598 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:42.541503    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:42.542365 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:42.542263    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:36:56.690711 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.690673    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"]
Sep 04 22:36:56.693403 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.693379    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.702802 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.702777    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2df128\"/\"ipp-route-2df128-kserve-self-signed-certs\""
Sep 04 22:36:56.702918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.702785    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2df128\"/\"kube-root-ca.crt\""
Sep 04 22:36:56.702918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.702842    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-2df128\"/\"openshift-service-ca.crt\""
Sep 04 22:36:56.702918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.702848    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-2df128\"/\"default-dockercfg-k26kw\""
Sep 04 22:36:56.717149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.717118    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"]
Sep 04 22:36:56.744745 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.744707    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:36:56.780694 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.780650    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5d9\" (UniqueName: \"kubernetes.io/projected/bc335545-ba57-4ac1-9347-31558a28351e-kube-api-access-px5d9\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.780888 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.780791    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-dshm\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.780888 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.780869    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-tmp-dir\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.781004 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.780909    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc335545-ba57-4ac1-9347-31558a28351e-tls-certs\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.781004 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.780948    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-home\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.781144 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.781024    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-model-cache\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.881898 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.881860    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-dshm\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.882116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.881923    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-tmp-dir\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.882116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.881949    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc335545-ba57-4ac1-9347-31558a28351e-tls-certs\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.882116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.881979    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-home\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.882116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.882013    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-model-cache\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.882116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.882073    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5d9\" (UniqueName: \"kubernetes.io/projected/bc335545-ba57-4ac1-9347-31558a28351e-kube-api-access-px5d9\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.882481 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.882453    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-tmp-dir\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.882571 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.882447    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-model-cache\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.882725 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.882698    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-home\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.884491 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.884471    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-dshm\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.884779 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.884758    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc335545-ba57-4ac1-9347-31558a28351e-tls-certs\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:56.892735 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:56.892709    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5d9\" (UniqueName: \"kubernetes.io/projected/bc335545-ba57-4ac1-9347-31558a28351e-kube-api-access-px5d9\") pod \"ipp-route-2df128-kserve-69fcc8dbb7-ttrbm\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") " pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:57.002864 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:57.002824    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:36:57.280298 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:57.280270    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"]
Sep 04 22:36:57.282035 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:36:57.282004    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc335545_ba57_4ac1_9347_31558a28351e.slice/crio-c8df67297bd9006e1b3b263b6ac9edf84762bb59bbd1a3572ebbc8c46913a960 WatchSource:0}: Error finding container c8df67297bd9006e1b3b263b6ac9edf84762bb59bbd1a3572ebbc8c46913a960: Status 404 returned error can't find the container with id c8df67297bd9006e1b3b263b6ac9edf84762bb59bbd1a3572ebbc8c46913a960
Sep 04 22:36:57.705890 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:57.705853    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm" event={"ID":"bc335545-ba57-4ac1-9347-31558a28351e","Type":"ContainerStarted","Data":"0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4"}
Sep 04 22:36:57.706430 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:57.705902    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm" event={"ID":"bc335545-ba57-4ac1-9347-31558a28351e","Type":"ContainerStarted","Data":"c8df67297bd9006e1b3b263b6ac9edf84762bb59bbd1a3572ebbc8c46913a960"}
Sep 04 22:36:57.726971 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:36:57.726912    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm" podStartSLOduration=1.72689562 podStartE2EDuration="1.72689562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:36:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:36:57.725844633 +0000 UTC m=+1701.244567178" watchObservedRunningTime="2026-09-04 22:36:57.72689562 +0000 UTC m=+1701.245618153"
Sep 04 22:37:01.417134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:01.417031    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:37:07.003385 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:07.003343    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:37:07.003866 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:07.003399    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:37:07.019166 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:07.019135    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:37:07.772408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:07.772375    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:37:11.272534 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:11.272476    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:13.614032 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:13.613998    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:37:21.006109 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:21.006067    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:21.345288 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:21.345194    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:21.981787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:21.981734    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:24.535985 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:24.535950    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:24.635156 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:24.635118    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:25.659948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.659900    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"]
Sep 04 22:37:25.662670 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.662649    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.665699 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.665673    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-919fd8-serving-cert\""
Sep 04 22:37:25.676526 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.676492    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"]
Sep 04 22:37:25.717787 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.717755    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:25.782204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.782172    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.782373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.782212    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.782373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.782255    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa4736-c768-4ef3-997f-50939d1cb163-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.782373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.782289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2pjs\" (UniqueName: \"kubernetes.io/projected/fbaa4736-c768-4ef3-997f-50939d1cb163-kube-api-access-t2pjs\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.883496 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.883454    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.883678 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.883539    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa4736-c768-4ef3-997f-50939d1cb163-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.883678 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:37:25.883592    2576 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-serving-cert: secret "maas-api-e2e-ipp-cleanup-919fd8-serving-cert" not found
Sep 04 22:37:25.883678 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:37:25.883668    2576 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-maas-api-tls podName:fbaa4736-c768-4ef3-997f-50939d1cb163 nodeName:}" failed. No retries permitted until 2026-09-04 22:37:26.383649925 +0000 UTC m=+1729.902372436 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-maas-api-tls") pod "maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq" (UID: "fbaa4736-c768-4ef3-997f-50939d1cb163") : secret "maas-api-e2e-ipp-cleanup-919fd8-serving-cert" not found
Sep 04 22:37:25.883678 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.883597    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2pjs\" (UniqueName: \"kubernetes.io/projected/fbaa4736-c768-4ef3-997f-50939d1cb163-kube-api-access-t2pjs\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.883828 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.883807    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.884241 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.884215    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa4736-c768-4ef3-997f-50939d1cb163-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.886168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.886142    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:25.888432 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.888409    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:25.894256 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:25.894234    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2pjs\" (UniqueName: \"kubernetes.io/projected/fbaa4736-c768-4ef3-997f-50939d1cb163-kube-api-access-t2pjs\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:26.354940 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:26.354898    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:26.389213 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:26.389173    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:26.391724 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:26.391700    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:26.483513 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:26.483474    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:26.577847 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:26.577804    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:26.579246 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:26.579205    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:26.853517 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:26.853488    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"]
Sep 04 22:37:26.855847 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:37:26.855822    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbaa4736_c768_4ef3_997f_50939d1cb163.slice/crio-5eebd52a075b96831158f322c9c0d21e550524d46101fbd54fec969b28e248ca WatchSource:0}: Error finding container 5eebd52a075b96831158f322c9c0d21e550524d46101fbd54fec969b28e248ca: Status 404 returned error can't find the container with id 5eebd52a075b96831158f322c9c0d21e550524d46101fbd54fec969b28e248ca
Sep 04 22:37:27.866969 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:27.866932    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq" event={"ID":"fbaa4736-c768-4ef3-997f-50939d1cb163","Type":"ContainerStarted","Data":"3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0"}
Sep 04 22:37:27.867408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:27.866992    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq" event={"ID":"fbaa4736-c768-4ef3-997f-50939d1cb163","Type":"ContainerStarted","Data":"5eebd52a075b96831158f322c9c0d21e550524d46101fbd54fec969b28e248ca"}
Sep 04 22:37:27.867408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:27.867022    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:27.889597 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:27.889544    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq" podStartSLOduration=2.540242185 podStartE2EDuration="2.889529321s" totalImagesPullingTime="349.287136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:37:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:37:27.887584339 +0000 UTC m=+1731.406306885" watchObservedRunningTime="2026-09-04 22:37:27.889529321 +0000 UTC m=+1731.408251885"
Sep 04 22:37:30.813042 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:30.813006    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:37:33.878134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:33.878104    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:49.544632 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:49.544596    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:56.366695 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.366647    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"]
Sep 04 22:37:56.367127 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.366924    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq" podUID="fbaa4736-c768-4ef3-997f-50939d1cb163" containerName="maas-api" containerID="cri-o://3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0" gracePeriod=30
Sep 04 22:37:56.408532 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.408490    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:56.507824 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.507780    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:56.613977 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.613942    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:56.626204 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.626146    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:56.696844 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.696810    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:56.708928 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.708901    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-maas-api-tls\") pod \"fbaa4736-c768-4ef3-997f-50939d1cb163\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") "
Sep 04 22:37:56.709118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.708953    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-metrics-certs\") pod \"fbaa4736-c768-4ef3-997f-50939d1cb163\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") "
Sep 04 22:37:56.709118 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.709100    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa4736-c768-4ef3-997f-50939d1cb163-openshift-service-ca\") pod \"fbaa4736-c768-4ef3-997f-50939d1cb163\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") "
Sep 04 22:37:56.709232 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.709119    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2pjs\" (UniqueName: \"kubernetes.io/projected/fbaa4736-c768-4ef3-997f-50939d1cb163-kube-api-access-t2pjs\") pod \"fbaa4736-c768-4ef3-997f-50939d1cb163\" (UID: \"fbaa4736-c768-4ef3-997f-50939d1cb163\") "
Sep 04 22:37:56.709503 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.709473    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbaa4736-c768-4ef3-997f-50939d1cb163-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fbaa4736-c768-4ef3-997f-50939d1cb163" (UID: "fbaa4736-c768-4ef3-997f-50939d1cb163"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:37:56.711121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.711091    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "fbaa4736-c768-4ef3-997f-50939d1cb163" (UID: "fbaa4736-c768-4ef3-997f-50939d1cb163"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:37:56.711229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.711156    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbaa4736-c768-4ef3-997f-50939d1cb163-kube-api-access-t2pjs" (OuterVolumeSpecName: "kube-api-access-t2pjs") pod "fbaa4736-c768-4ef3-997f-50939d1cb163" (UID: "fbaa4736-c768-4ef3-997f-50939d1cb163"). InnerVolumeSpecName "kube-api-access-t2pjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:37:56.711229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.711188    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fbaa4736-c768-4ef3-997f-50939d1cb163" (UID: "fbaa4736-c768-4ef3-997f-50939d1cb163"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:37:56.809938 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.809895    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fbaa4736-c768-4ef3-997f-50939d1cb163-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:37:56.809938 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.809929    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2pjs\" (UniqueName: \"kubernetes.io/projected/fbaa4736-c768-4ef3-997f-50939d1cb163-kube-api-access-t2pjs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:37:56.809938 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.809941    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:37:56.810213 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:56.809950    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbaa4736-c768-4ef3-997f-50939d1cb163-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:37:57.022722 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.022676    2576 generic.go:356] "Generic (PLEG): container finished" podID="fbaa4736-c768-4ef3-997f-50939d1cb163" containerID="3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0" exitCode=0
Sep 04 22:37:57.022904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.022747    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"
Sep 04 22:37:57.022904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.022764    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq" event={"ID":"fbaa4736-c768-4ef3-997f-50939d1cb163","Type":"ContainerDied","Data":"3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0"}
Sep 04 22:37:57.022904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.022816    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq" event={"ID":"fbaa4736-c768-4ef3-997f-50939d1cb163","Type":"ContainerDied","Data":"5eebd52a075b96831158f322c9c0d21e550524d46101fbd54fec969b28e248ca"}
Sep 04 22:37:57.022904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.022843    2576 scope.go:117] "RemoveContainer" containerID="3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0"
Sep 04 22:37:57.032352 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.032332    2576 scope.go:117] "RemoveContainer" containerID="3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0"
Sep 04 22:37:57.032693 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:37:57.032667    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0\": container with ID starting with 3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0 not found: ID does not exist" containerID="3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0"
Sep 04 22:37:57.032742 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.032702    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0"} err="failed to get container status \"3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0\": rpc error: code = NotFound desc = could not find container \"3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0\": container with ID starting with 3f9ec2a2cad47c600c05f675da1d1aa17830c974a43bb923589233a56dc681f0 not found: ID does not exist"
Sep 04 22:37:57.044715 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.044679    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"]
Sep 04 22:37:57.048656 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.048625    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-919fd8-856f8cfccd-4npbq"]
Sep 04 22:37:57.079531 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:57.079498    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:37:58.919644 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:37:58.919611    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fbaa4736-c768-4ef3-997f-50939d1cb163" path="/var/lib/kubelet/pods/fbaa4736-c768-4ef3-997f-50939d1cb163/volumes"
Sep 04 22:38:00.609407 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:00.609367    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:38:03.444413 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:03.444379    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:38:05.141543 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:05.141499    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:38:05.240283 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:05.240245    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:38:07.614548 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:07.614510    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:38:23.005420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.005384    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"]
Sep 04 22:38:23.005959 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.005597    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm" podUID="68c88a21-4797-4c81-b408-b84b3f359957" containerName="maas-api" containerID="cri-o://2671bc997c1333376e1c3d2d4241ff7eb74ac64363789a05e1316b3b0efa8539" gracePeriod=30
Sep 04 22:38:23.167181 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.167146    2576 generic.go:356] "Generic (PLEG): container finished" podID="68c88a21-4797-4c81-b408-b84b3f359957" containerID="2671bc997c1333376e1c3d2d4241ff7eb74ac64363789a05e1316b3b0efa8539" exitCode=0
Sep 04 22:38:23.167339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.167196    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm" event={"ID":"68c88a21-4797-4c81-b408-b84b3f359957","Type":"ContainerDied","Data":"2671bc997c1333376e1c3d2d4241ff7eb74ac64363789a05e1316b3b0efa8539"}
Sep 04 22:38:23.253920 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.253883    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:38:23.280487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.280409    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsdcn\" (UniqueName: \"kubernetes.io/projected/68c88a21-4797-4c81-b408-b84b3f359957-kube-api-access-tsdcn\") pod \"68c88a21-4797-4c81-b408-b84b3f359957\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") "
Sep 04 22:38:23.280487 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.280464    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-maas-api-tls\") pod \"68c88a21-4797-4c81-b408-b84b3f359957\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") "
Sep 04 22:38:23.280705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.280560    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-metrics-certs\") pod \"68c88a21-4797-4c81-b408-b84b3f359957\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") "
Sep 04 22:38:23.280705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.280593    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c88a21-4797-4c81-b408-b84b3f359957-openshift-service-ca\") pod \"68c88a21-4797-4c81-b408-b84b3f359957\" (UID: \"68c88a21-4797-4c81-b408-b84b3f359957\") "
Sep 04 22:38:23.280994 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.280966    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68c88a21-4797-4c81-b408-b84b3f359957-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "68c88a21-4797-4c81-b408-b84b3f359957" (UID: "68c88a21-4797-4c81-b408-b84b3f359957"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:38:23.282692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.282663    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "68c88a21-4797-4c81-b408-b84b3f359957" (UID: "68c88a21-4797-4c81-b408-b84b3f359957"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:38:23.282799 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.282691    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "68c88a21-4797-4c81-b408-b84b3f359957" (UID: "68c88a21-4797-4c81-b408-b84b3f359957"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:38:23.282799 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.282781    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68c88a21-4797-4c81-b408-b84b3f359957-kube-api-access-tsdcn" (OuterVolumeSpecName: "kube-api-access-tsdcn") pod "68c88a21-4797-4c81-b408-b84b3f359957" (UID: "68c88a21-4797-4c81-b408-b84b3f359957"). InnerVolumeSpecName "kube-api-access-tsdcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:38:23.381879 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.381842    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68c88a21-4797-4c81-b408-b84b3f359957-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:23.381879 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.381873    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsdcn\" (UniqueName: \"kubernetes.io/projected/68c88a21-4797-4c81-b408-b84b3f359957-kube-api-access-tsdcn\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:23.381879 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.381885    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:23.382134 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.381894    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68c88a21-4797-4c81-b408-b84b3f359957-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:23.619249 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:23.619155    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:38:24.148704 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:24.148668    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:38:24.173600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:24.173567    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm" event={"ID":"68c88a21-4797-4c81-b408-b84b3f359957","Type":"ContainerDied","Data":"3dc61736d25819d46f93ed8259981733425d74349746b2ff6c0a3aa7edd8ee1b"}
Sep 04 22:38:24.173805 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:24.173611    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"
Sep 04 22:38:24.173805 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:24.173630    2576 scope.go:117] "RemoveContainer" containerID="2671bc997c1333376e1c3d2d4241ff7eb74ac64363789a05e1316b3b0efa8539"
Sep 04 22:38:24.202323 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:24.202291    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"]
Sep 04 22:38:24.208420 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:24.208394    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-2df128-5689478c45-g6bfm"]
Sep 04 22:38:24.228527 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:24.228492    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:38:24.918125 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:24.918093    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68c88a21-4797-4c81-b408-b84b3f359957" path="/var/lib/kubelet/pods/68c88a21-4797-4c81-b408-b84b3f359957/volumes"
Sep 04 22:38:28.724852 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:28.724806    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"]
Sep 04 22:38:28.819595 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:28.819557    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:38:33.501510 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:33.501467    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:38:37.030610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:37.030573    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:38:37.033797 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:37.033763    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:38:43.744747 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:43.744695    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm" podUID="bc335545-ba57-4ac1-9347-31558a28351e" containerName="main" containerID="cri-o://0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4" gracePeriod=45
Sep 04 22:38:44.009142 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.009012    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:38:44.186246 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186211    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-tmp-dir\") pod \"bc335545-ba57-4ac1-9347-31558a28351e\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") "
Sep 04 22:38:44.186450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186289    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-model-cache\") pod \"bc335545-ba57-4ac1-9347-31558a28351e\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") "
Sep 04 22:38:44.186450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186349    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px5d9\" (UniqueName: \"kubernetes.io/projected/bc335545-ba57-4ac1-9347-31558a28351e-kube-api-access-px5d9\") pod \"bc335545-ba57-4ac1-9347-31558a28351e\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") "
Sep 04 22:38:44.186450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186401    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-home\") pod \"bc335545-ba57-4ac1-9347-31558a28351e\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") "
Sep 04 22:38:44.186450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186426    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-dshm\") pod \"bc335545-ba57-4ac1-9347-31558a28351e\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") "
Sep 04 22:38:44.186666 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186460    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc335545-ba57-4ac1-9347-31558a28351e-tls-certs\") pod \"bc335545-ba57-4ac1-9347-31558a28351e\" (UID: \"bc335545-ba57-4ac1-9347-31558a28351e\") "
Sep 04 22:38:44.186666 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186527    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc335545-ba57-4ac1-9347-31558a28351e" (UID: "bc335545-ba57-4ac1-9347-31558a28351e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:38:44.186666 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186537    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc335545-ba57-4ac1-9347-31558a28351e" (UID: "bc335545-ba57-4ac1-9347-31558a28351e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:38:44.186817 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186689    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-home" (OuterVolumeSpecName: "home") pod "bc335545-ba57-4ac1-9347-31558a28351e" (UID: "bc335545-ba57-4ac1-9347-31558a28351e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:38:44.186887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186830    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:44.186887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186845    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:44.186887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.186854    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:44.188739 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.188712    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-dshm" (OuterVolumeSpecName: "dshm") pod "bc335545-ba57-4ac1-9347-31558a28351e" (UID: "bc335545-ba57-4ac1-9347-31558a28351e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:38:44.188864 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.188804    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc335545-ba57-4ac1-9347-31558a28351e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc335545-ba57-4ac1-9347-31558a28351e" (UID: "bc335545-ba57-4ac1-9347-31558a28351e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:38:44.188864 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.188816    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc335545-ba57-4ac1-9347-31558a28351e-kube-api-access-px5d9" (OuterVolumeSpecName: "kube-api-access-px5d9") pod "bc335545-ba57-4ac1-9347-31558a28351e" (UID: "bc335545-ba57-4ac1-9347-31558a28351e"). InnerVolumeSpecName "kube-api-access-px5d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:38:44.282546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.282447    2576 generic.go:356] "Generic (PLEG): container finished" podID="bc335545-ba57-4ac1-9347-31558a28351e" containerID="0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4" exitCode=0
Sep 04 22:38:44.282546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.282487    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"
Sep 04 22:38:44.282546 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.282535    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm" event={"ID":"bc335545-ba57-4ac1-9347-31558a28351e","Type":"ContainerDied","Data":"0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4"}
Sep 04 22:38:44.282846 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.282579    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm" event={"ID":"bc335545-ba57-4ac1-9347-31558a28351e","Type":"ContainerDied","Data":"c8df67297bd9006e1b3b263b6ac9edf84762bb59bbd1a3572ebbc8c46913a960"}
Sep 04 22:38:44.282846 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.282597    2576 scope.go:117] "RemoveContainer" containerID="0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4"
Sep 04 22:38:44.287533 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.287507    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5d9\" (UniqueName: \"kubernetes.io/projected/bc335545-ba57-4ac1-9347-31558a28351e-kube-api-access-px5d9\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:44.287661 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.287537    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc335545-ba57-4ac1-9347-31558a28351e-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:44.287661 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.287553    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc335545-ba57-4ac1-9347-31558a28351e-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:38:44.292885 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.292867    2576 scope.go:117] "RemoveContainer" containerID="0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4"
Sep 04 22:38:44.293259 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:38:44.293234    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4\": container with ID starting with 0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4 not found: ID does not exist" containerID="0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4"
Sep 04 22:38:44.293361 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.293268    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4"} err="failed to get container status \"0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4\": rpc error: code = NotFound desc = could not find container \"0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4\": container with ID starting with 0d6a80932de477419a97a3d88c5063586edee7a856cf897a6f80f3fd6355eec4 not found: ID does not exist"
Sep 04 22:38:44.307222 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.307193    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"]
Sep 04 22:38:44.312268 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.312242    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-2df128/ipp-route-2df128-kserve-69fcc8dbb7-ttrbm"]
Sep 04 22:38:44.919349 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:38:44.919312    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc335545-ba57-4ac1-9347-31558a28351e" path="/var/lib/kubelet/pods/bc335545-ba57-4ac1-9347-31558a28351e/volumes"
Sep 04 22:39:05.300541 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:05.300498    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:06.670662 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:06.670606    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:09.456428 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:09.456388    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:13.047478 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:13.047437    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:13.141318 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:13.141277    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:13.375964 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:13.375870    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:13.858909 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:13.858874    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:24.402307 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.402269    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:24.444358 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.444319    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"]
Sep 04 22:39:24.444626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.444577    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5" podUID="ea988877-6d14-4178-b514-982b54801708" containerName="payload-pre-processing" containerID="cri-o://b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d" gracePeriod=30
Sep 04 22:39:24.501048 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.501019    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:24.607355 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.607308    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:24.694539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.694508    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:24.703290 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.703269    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:39:24.791333 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.791294    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea988877-6d14-4178-b514-982b54801708-plugins-config-volume\") pod \"ea988877-6d14-4178-b514-982b54801708\" (UID: \"ea988877-6d14-4178-b514-982b54801708\") "
Sep 04 22:39:24.791506 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.791377    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhvcg\" (UniqueName: \"kubernetes.io/projected/ea988877-6d14-4178-b514-982b54801708-kube-api-access-lhvcg\") pod \"ea988877-6d14-4178-b514-982b54801708\" (UID: \"ea988877-6d14-4178-b514-982b54801708\") "
Sep 04 22:39:24.791638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.791607    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea988877-6d14-4178-b514-982b54801708-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea988877-6d14-4178-b514-982b54801708" (UID: "ea988877-6d14-4178-b514-982b54801708"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:39:24.791845 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.791827    2576 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea988877-6d14-4178-b514-982b54801708-plugins-config-volume\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:24.793395 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.793371    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea988877-6d14-4178-b514-982b54801708-kube-api-access-lhvcg" (OuterVolumeSpecName: "kube-api-access-lhvcg") pod "ea988877-6d14-4178-b514-982b54801708" (UID: "ea988877-6d14-4178-b514-982b54801708"). InnerVolumeSpecName "kube-api-access-lhvcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:39:24.893114 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:24.893078    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhvcg\" (UniqueName: \"kubernetes.io/projected/ea988877-6d14-4178-b514-982b54801708-kube-api-access-lhvcg\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:25.045966 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.045928    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:25.503522 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.503486    2576 generic.go:356] "Generic (PLEG): container finished" podID="ea988877-6d14-4178-b514-982b54801708" containerID="b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d" exitCode=0
Sep 04 22:39:25.503948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.503529    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"
Sep 04 22:39:25.503948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.503573    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5" event={"ID":"ea988877-6d14-4178-b514-982b54801708","Type":"ContainerDied","Data":"b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d"}
Sep 04 22:39:25.503948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.503621    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5" event={"ID":"ea988877-6d14-4178-b514-982b54801708","Type":"ContainerDied","Data":"4b3abf2e950857d81f1b5f671e12d273b46e36f24d3646a4b85cd6c499e73ce4"}
Sep 04 22:39:25.503948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.503651    2576 scope.go:117] "RemoveContainer" containerID="b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d"
Sep 04 22:39:25.512507 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.512489    2576 scope.go:117] "RemoveContainer" containerID="b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d"
Sep 04 22:39:25.512780 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:39:25.512761    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d\": container with ID starting with b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d not found: ID does not exist" containerID="b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d"
Sep 04 22:39:25.512840 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.512788    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d"} err="failed to get container status \"b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d\": rpc error: code = NotFound desc = could not find container \"b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d\": container with ID starting with b68adc1a8191ff9b9cddb14b030a84a51ad6ca364e4ab6fd32565a0ccbadfb2d not found: ID does not exist"
Sep 04 22:39:25.523735 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.523709    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"]
Sep 04 22:39:25.528843 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:25.528823    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ecfcad-96b554dd6-j24v5"]
Sep 04 22:39:26.920349 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:26.920311    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea988877-6d14-4178-b514-982b54801708" path="/var/lib/kubelet/pods/ea988877-6d14-4178-b514-982b54801708/volumes"
Sep 04 22:39:28.808226 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:28.808189    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:39:31.509660 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:31.509628    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:31.620121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:31.620079    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:31.772272 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:31.772181    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"]
Sep 04 22:39:31.772456 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:31.772411    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy" containerID="cri-o://8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11" gracePeriod=30
Sep 04 22:39:31.805676 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:31.805643    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:32.284098 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:32.284032    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:37.029442 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.029414    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:39:37.112941 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.112851    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-credential-socket\") pod \"001de2e3-d82e-4072-94f3-da88c216f903\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") "
Sep 04 22:39:37.112941 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.112931    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001de2e3-d82e-4072-94f3-da88c216f903-istiod-ca-cert\") pod \"001de2e3-d82e-4072-94f3-da88c216f903\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") "
Sep 04 22:39:37.113170 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.112977    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-certs\") pod \"001de2e3-d82e-4072-94f3-da88c216f903\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") "
Sep 04 22:39:37.113170 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113014    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-socket\") pod \"001de2e3-d82e-4072-94f3-da88c216f903\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") "
Sep 04 22:39:37.113170 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113140    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bvbpv\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-kube-api-access-bvbpv\") pod \"001de2e3-d82e-4072-94f3-da88c216f903\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") "
Sep 04 22:39:37.113330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113193    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001de2e3-d82e-4072-94f3-da88c216f903-istio-podinfo\") pod \"001de2e3-d82e-4072-94f3-da88c216f903\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") "
Sep 04 22:39:37.113330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113224    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "001de2e3-d82e-4072-94f3-da88c216f903" (UID: "001de2e3-d82e-4072-94f3-da88c216f903"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:37.113330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113277    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-data\") pod \"001de2e3-d82e-4072-94f3-da88c216f903\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") "
Sep 04 22:39:37.113330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113309    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "001de2e3-d82e-4072-94f3-da88c216f903" (UID: "001de2e3-d82e-4072-94f3-da88c216f903"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:37.113330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113301    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "001de2e3-d82e-4072-94f3-da88c216f903" (UID: "001de2e3-d82e-4072-94f3-da88c216f903"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:37.113604 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113338    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-istio-token\") pod \"001de2e3-d82e-4072-94f3-da88c216f903\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") "
Sep 04 22:39:37.113604 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113364    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-envoy\") pod \"001de2e3-d82e-4072-94f3-da88c216f903\" (UID: \"001de2e3-d82e-4072-94f3-da88c216f903\") "
Sep 04 22:39:37.113604 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113412    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/001de2e3-d82e-4072-94f3-da88c216f903-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "001de2e3-d82e-4072-94f3-da88c216f903" (UID: "001de2e3-d82e-4072-94f3-da88c216f903"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:39:37.113604 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113569    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-data" (OuterVolumeSpecName: "istio-data") pod "001de2e3-d82e-4072-94f3-da88c216f903" (UID: "001de2e3-d82e-4072-94f3-da88c216f903"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:37.113801 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113721    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/001de2e3-d82e-4072-94f3-da88c216f903-istiod-ca-cert\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:37.113801 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113739    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:37.113801 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113750    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-workload-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:37.113801 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113762    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-data\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:37.113801 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.113775    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-credential-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:37.115445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.115419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "001de2e3-d82e-4072-94f3-da88c216f903" (UID: "001de2e3-d82e-4072-94f3-da88c216f903"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:37.115560 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.115500    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/001de2e3-d82e-4072-94f3-da88c216f903-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "001de2e3-d82e-4072-94f3-da88c216f903" (UID: "001de2e3-d82e-4072-94f3-da88c216f903"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 22:39:37.115560 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.115514    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-istio-token" (OuterVolumeSpecName: "istio-token") pod "001de2e3-d82e-4072-94f3-da88c216f903" (UID: "001de2e3-d82e-4072-94f3-da88c216f903"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:39:37.115560 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.115533    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-kube-api-access-bvbpv" (OuterVolumeSpecName: "kube-api-access-bvbpv") pod "001de2e3-d82e-4072-94f3-da88c216f903" (UID: "001de2e3-d82e-4072-94f3-da88c216f903"). InnerVolumeSpecName "kube-api-access-bvbpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:39:37.214887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.214852    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvbpv\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-kube-api-access-bvbpv\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:37.214887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.214884    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/001de2e3-d82e-4072-94f3-da88c216f903-istio-podinfo\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:37.214887 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.214893    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/001de2e3-d82e-4072-94f3-da88c216f903-istio-token\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:37.215145 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.214903    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/001de2e3-d82e-4072-94f3-da88c216f903-istio-envoy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:37.568845 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.568805    2576 generic.go:356] "Generic (PLEG): container finished" podID="001de2e3-d82e-4072-94f3-da88c216f903" containerID="8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11" exitCode=0
Sep 04 22:39:37.569027 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.568861    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"
Sep 04 22:39:37.569027 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.568892    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" event={"ID":"001de2e3-d82e-4072-94f3-da88c216f903","Type":"ContainerDied","Data":"8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11"}
Sep 04 22:39:37.569027 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.568950    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24" event={"ID":"001de2e3-d82e-4072-94f3-da88c216f903","Type":"ContainerDied","Data":"61ab714e926c164cda90021f91d3ffd7a8e4632f453c9e2eea135816d77f1b64"}
Sep 04 22:39:37.569027 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.568974    2576 scope.go:117] "RemoveContainer" containerID="8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11"
Sep 04 22:39:37.578165 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.578143    2576 scope.go:117] "RemoveContainer" containerID="b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef"
Sep 04 22:39:37.585706 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.585689    2576 scope.go:117] "RemoveContainer" containerID="8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11"
Sep 04 22:39:37.585925 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:39:37.585909    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11\": container with ID starting with 8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11 not found: ID does not exist" containerID="8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11"
Sep 04 22:39:37.585989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.585933    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11"} err="failed to get container status \"8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11\": rpc error: code = NotFound desc = could not find container \"8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11\": container with ID starting with 8eefa2c2c9f46422567e1d3f1cabbe60cf5ee924b6f52aeaa329ca4a11d27a11 not found: ID does not exist"
Sep 04 22:39:37.585989 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.585947    2576 scope.go:117] "RemoveContainer" containerID="b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef"
Sep 04 22:39:37.586214 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:39:37.586199    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef\": container with ID starting with b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef not found: ID does not exist" containerID="b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef"
Sep 04 22:39:37.586256 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.586217    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef"} err="failed to get container status \"b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef\": rpc error: code = NotFound desc = could not find container \"b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef\": container with ID starting with b6be1ae6841c10c074c0ab8e1d71a017a6318ab128c866d51ec2027663c500ef not found: ID does not exist"
Sep 04 22:39:37.593759 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.593734    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"]
Sep 04 22:39:37.602949 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:37.602928    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-ecfcad-openshift-default-5f859b5c9b-lrf24"]
Sep 04 22:39:38.149893 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:38.149856    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:38.918624 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:38.918590    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="001de2e3-d82e-4072-94f3-da88c216f903" path="/var/lib/kubelet/pods/001de2e3-d82e-4072-94f3-da88c216f903/volumes"
Sep 04 22:39:42.415379 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:42.415339    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:43.150173 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:43.150133    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:43.240381 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:43.240316    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:43.333271 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:43.333230    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:43.434210 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:43.434106    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:43.514633 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:43.514601    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:43.798348 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:43.798315    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:46.711149 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:46.711116    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:39:50.493813 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:50.493767    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:50.590706 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:50.590655    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:50.772815 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:50.772730    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"]
Sep 04 22:39:50.773014 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:50.772990    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy" containerID="cri-o://139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be" gracePeriod=30
Sep 04 22:39:51.198600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:51.198514    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:51.287618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:51.287559    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:39:52.075472 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:52.075431    2576 patch_prober.go:28] interesting pod/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 04 22:39:52.075920 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:52.075492    2576 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 04 22:39:56.018014 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.017984    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:39:56.103522 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103436    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-envoy\") pod \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") "
Sep 04 22:39:56.103522 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103489    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g8m5s\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-kube-api-access-g8m5s\") pod \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") "
Sep 04 22:39:56.103522 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103511    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-data\") pod \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") "
Sep 04 22:39:56.103814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103539    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-socket\") pod \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") "
Sep 04 22:39:56.103814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103579    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-token\") pod \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") "
Sep 04 22:39:56.103814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103600    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-podinfo\") pod \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") "
Sep 04 22:39:56.103814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103633    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istiod-ca-cert\") pod \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") "
Sep 04 22:39:56.103814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103700    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-certs\") pod \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") "
Sep 04 22:39:56.103814 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103731    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-credential-socket\") pod \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\" (UID: \"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb\") "
Sep 04 22:39:56.104153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103834    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-data" (OuterVolumeSpecName: "istio-data") pod "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" (UID: "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:56.104153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.103881    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" (UID: "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:56.104153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.104027    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" (UID: "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:56.104153 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.104093    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" (UID: "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:39:56.104371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.104163    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" (UID: "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:56.104371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.104203    2576 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-credential-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:56.104371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.104222    2576 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-data\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:56.104371 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.104235    2576 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-socket\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:56.105871 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.105839    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-kube-api-access-g8m5s" (OuterVolumeSpecName: "kube-api-access-g8m5s") pod "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" (UID: "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb"). InnerVolumeSpecName "kube-api-access-g8m5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:39:56.105871 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.105842    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" (UID: "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:39:56.106026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.105951    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" (UID: "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 04 22:39:56.106026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.105972    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-token" (OuterVolumeSpecName: "istio-token") pod "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" (UID: "43cb9cfb-f543-4af3-96e4-1aa7bafe65fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:39:56.205140 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.205107    2576 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-token\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:56.205140 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.205136    2576 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-podinfo\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:56.205140 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.205146    2576 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istiod-ca-cert\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:56.205373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.205155    2576 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-workload-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:56.205373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.205163    2576 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-istio-envoy\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:56.205373 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.205171    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8m5s\" (UniqueName: \"kubernetes.io/projected/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb-kube-api-access-g8m5s\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:39:56.670605 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.670568    2576 generic.go:356] "Generic (PLEG): container finished" podID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerID="139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be" exitCode=0
Sep 04 22:39:56.670831 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.670626    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" event={"ID":"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb","Type":"ContainerDied","Data":"139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be"}
Sep 04 22:39:56.670831 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.670631    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"
Sep 04 22:39:56.670831 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.670715    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs" event={"ID":"43cb9cfb-f543-4af3-96e4-1aa7bafe65fb","Type":"ContainerDied","Data":"25f7f14049688b43470bb04373bc445d57ac33b55c793f82f7e5d571aa0d7481"}
Sep 04 22:39:56.670831 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.670747    2576 scope.go:117] "RemoveContainer" containerID="139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be"
Sep 04 22:39:56.679859 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.679841    2576 scope.go:117] "RemoveContainer" containerID="18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd"
Sep 04 22:39:56.688331 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.688314    2576 scope.go:117] "RemoveContainer" containerID="139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be"
Sep 04 22:39:56.688603 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:39:56.688579    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be\": container with ID starting with 139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be not found: ID does not exist" containerID="139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be"
Sep 04 22:39:56.688675 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.688611    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be"} err="failed to get container status \"139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be\": rpc error: code = NotFound desc = could not find container \"139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be\": container with ID starting with 139b861b57824ebaa88c6e7b92447803ef8bd34208b5f06e5896b83a561823be not found: ID does not exist"
Sep 04 22:39:56.688675 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.688628    2576 scope.go:117] "RemoveContainer" containerID="18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd"
Sep 04 22:39:56.688872 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:39:56.688856    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd\": container with ID starting with 18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd not found: ID does not exist" containerID="18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd"
Sep 04 22:39:56.688913 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.688877    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd"} err="failed to get container status \"18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd\": rpc error: code = NotFound desc = could not find container \"18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd\": container with ID starting with 18e03d162182e31cd2af1a2131dbc98d6f9fd5dd6b97b5a1d979275861c420bd not found: ID does not exist"
Sep 04 22:39:56.698857 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.698829    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"]
Sep 04 22:39:56.705834 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.705810    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-bc62f0-openshift-default-6b74968974-jkmrs"]
Sep 04 22:39:56.918737 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:39:56.918700    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" path="/var/lib/kubelet/pods/43cb9cfb-f543-4af3-96e4-1aa7bafe65fb/volumes"
Sep 04 22:40:01.763556 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:01.763479    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"]
Sep 04 22:40:01.763954 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:01.763694    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4" podUID="bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" containerName="maas-api" containerID="cri-o://478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8" gracePeriod=30
Sep 04 22:40:02.012898 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.012873    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:40:02.118117 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.118016    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:40:02.168716 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.168679    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgvls\" (UniqueName: \"kubernetes.io/projected/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-kube-api-access-sgvls\") pod \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") "
Sep 04 22:40:02.168905 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.168760    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-openshift-service-ca\") pod \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") "
Sep 04 22:40:02.168905 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.168794    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-metrics-certs\") pod \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") "
Sep 04 22:40:02.168905 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.168824    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-maas-api-tls\") pod \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\" (UID: \"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e\") "
Sep 04 22:40:02.169229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.169201    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" (UID: "bc1b00a0-b8dd-4db4-a93d-7bb0c160560e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:40:02.171037 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.171007    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" (UID: "bc1b00a0-b8dd-4db4-a93d-7bb0c160560e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:40:02.171197 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.171036    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" (UID: "bc1b00a0-b8dd-4db4-a93d-7bb0c160560e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:40:02.171197 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.171045    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-kube-api-access-sgvls" (OuterVolumeSpecName: "kube-api-access-sgvls") pod "bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" (UID: "bc1b00a0-b8dd-4db4-a93d-7bb0c160560e"). InnerVolumeSpecName "kube-api-access-sgvls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:40:02.211613 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.211581    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:40:02.270470 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.270430    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:40:02.270470 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.270463    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:40:02.270470 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.270477    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:40:02.270722 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.270488    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgvls\" (UniqueName: \"kubernetes.io/projected/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e-kube-api-access-sgvls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:40:02.507670 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.507635    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:40:02.702689 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.702649    2576 generic.go:356] "Generic (PLEG): container finished" podID="bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" containerID="478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8" exitCode=0
Sep 04 22:40:02.702869 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.702703    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"
Sep 04 22:40:02.702869 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.702737    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4" event={"ID":"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e","Type":"ContainerDied","Data":"478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8"}
Sep 04 22:40:02.702869 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.702793    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4" event={"ID":"bc1b00a0-b8dd-4db4-a93d-7bb0c160560e","Type":"ContainerDied","Data":"27a71fbc92fbba6fc73a68fae9979d337bfcfd6d59e346b526aa8cc1b3b02e14"}
Sep 04 22:40:02.702869 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.702823    2576 scope.go:117] "RemoveContainer" containerID="478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8"
Sep 04 22:40:02.719271 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.719241    2576 scope.go:117] "RemoveContainer" containerID="478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8"
Sep 04 22:40:02.719604 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:40:02.719580    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8\": container with ID starting with 478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8 not found: ID does not exist" containerID="478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8"
Sep 04 22:40:02.719683 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.719613    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8"} err="failed to get container status \"478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8\": rpc error: code = NotFound desc = could not find container \"478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8\": container with ID starting with 478e523cf7f98da949f19853eeefd701acf522d355b5bbe3948f9804928446d8 not found: ID does not exist"
Sep 04 22:40:02.731040 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.731013    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"]
Sep 04 22:40:02.739700 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.739675    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-bf86db-6cff699896-gvvx4"]
Sep 04 22:40:02.918252 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:02.918174    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" path="/var/lib/kubelet/pods/bc1b00a0-b8dd-4db4-a93d-7bb0c160560e/volumes"
Sep 04 22:40:03.504549 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:03.504515    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:40:08.152315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:08.152284    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:40:10.147471 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:10.147433    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:40:10.269460 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:10.269428    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:40:13.911388 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:13.911349    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:40:16.659129 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:40:16.659013    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:41:34.904785 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:41:34.904747    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:41:36.076342 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:41:36.076299    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:41:37.625177 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:41:37.625142    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:42:02.937643 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:42:02.937597    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:42:08.691280 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:42:08.691249    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:42:11.047508 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:42:11.047468    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:42:18.410804 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:42:18.410773    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:42:25.401275 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:42:25.401239    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:42:38.160612 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:42:38.160576    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:42:48.211562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:42:48.211526    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:43:37.062886 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:43:37.062855    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:43:37.068600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:43:37.068576    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:44:15.620208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:15.620167    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:44:23.421838 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:23.421806    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:44:29.804229 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:29.804144    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:44:30.594569 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.594530    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"]
Sep 04 22:44:30.594976 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.594957    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68c88a21-4797-4c81-b408-b84b3f359957" containerName="maas-api"
Sep 04 22:44:30.595026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.594978    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="68c88a21-4797-4c81-b408-b84b3f359957" containerName="maas-api"
Sep 04 22:44:30.595026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.594987    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc335545-ba57-4ac1-9347-31558a28351e" containerName="main"
Sep 04 22:44:30.595026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.594992    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc335545-ba57-4ac1-9347-31558a28351e" containerName="main"
Sep 04 22:44:30.595026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.594999    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy"
Sep 04 22:44:30.595026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595004    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy"
Sep 04 22:44:30.595026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595013    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" containerName="maas-api"
Sep 04 22:44:30.595026 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595018    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" containerName="maas-api"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595035    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595041    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595048    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea988877-6d14-4178-b514-982b54801708" containerName="payload-pre-processing"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595068    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea988877-6d14-4178-b514-982b54801708" containerName="payload-pre-processing"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595074    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595079    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595085    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fbaa4736-c768-4ef3-997f-50939d1cb163" containerName="maas-api"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595089    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="fbaa4736-c768-4ef3-997f-50939d1cb163" containerName="maas-api"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595156    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1b00a0-b8dd-4db4-a93d-7bb0c160560e" containerName="maas-api"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595164    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea988877-6d14-4178-b514-982b54801708" containerName="payload-pre-processing"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595172    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595177    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="43cb9cfb-f543-4af3-96e4-1aa7bafe65fb" containerName="istio-proxy"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595182    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595188    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="fbaa4736-c768-4ef3-997f-50939d1cb163" containerName="maas-api"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595194    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="68c88a21-4797-4c81-b408-b84b3f359957" containerName="maas-api"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595201    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc335545-ba57-4ac1-9347-31558a28351e" containerName="main"
Sep 04 22:44:30.595262 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.595207    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy"
Sep 04 22:44:30.598308 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.598293    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.601071 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.601039    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c86c36be-kserve-self-signed-certs\""
Sep 04 22:44:30.608588 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.608563    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"]
Sep 04 22:44:30.635317 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.635289    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dbec85-4bad-4218-a877-3d8126dfacb7-tls-certs\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.635426 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.635348    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-tmp-dir\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.635426 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.635368    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-home\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.635426 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.635389    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7v5b\" (UniqueName: \"kubernetes.io/projected/b1dbec85-4bad-4218-a877-3d8126dfacb7-kube-api-access-p7v5b\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.635426 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.635415    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-dshm\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.635570 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.635453    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-model-cache\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.720872 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.720838    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:44:30.736829 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.736803    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-tmp-dir\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.736970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.736836    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-home\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.736970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.736859    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7v5b\" (UniqueName: \"kubernetes.io/projected/b1dbec85-4bad-4218-a877-3d8126dfacb7-kube-api-access-p7v5b\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.736970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.736880    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-dshm\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.736970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.736903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-model-cache\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.736970 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.736954    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dbec85-4bad-4218-a877-3d8126dfacb7-tls-certs\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.737283 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.737255    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-tmp-dir\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.737339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.737283    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-home\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.737388 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.737341    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-model-cache\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.739116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.739096    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-dshm\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.739431 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.739414    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dbec85-4bad-4218-a877-3d8126dfacb7-tls-certs\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.745812 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.745785    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7v5b\" (UniqueName: \"kubernetes.io/projected/b1dbec85-4bad-4218-a877-3d8126dfacb7-kube-api-access-p7v5b\") pod \"e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") " pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:30.907553 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:30.907464    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:31.180304 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:31.180278    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"]
Sep 04 22:44:31.182616 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:44:31.182590    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1dbec85_4bad_4218_a877_3d8126dfacb7.slice/crio-2dc9c391f5895feda17e06b500c1f15d180408c0587cd910f814980222de2620 WatchSource:0}: Error finding container 2dc9c391f5895feda17e06b500c1f15d180408c0587cd910f814980222de2620: Status 404 returned error can't find the container with id 2dc9c391f5895feda17e06b500c1f15d180408c0587cd910f814980222de2620
Sep 04 22:44:32.028941 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:32.028898    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk" event={"ID":"b1dbec85-4bad-4218-a877-3d8126dfacb7","Type":"ContainerStarted","Data":"3c9b7bc8e8f901cd4e44b09c493cc44e9def6a949c76c6a2865ca4274646ecf8"}
Sep 04 22:44:32.028941 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:32.028946    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk" event={"ID":"b1dbec85-4bad-4218-a877-3d8126dfacb7","Type":"ContainerStarted","Data":"2dc9c391f5895feda17e06b500c1f15d180408c0587cd910f814980222de2620"}
Sep 04 22:44:32.047626 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:32.047573    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk" podStartSLOduration=2.04753572 podStartE2EDuration="2.04753572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:44:32.046903853 +0000 UTC m=+2155.565626386" watchObservedRunningTime="2026-09-04 22:44:32.04753572 +0000 UTC m=+2155.566258253"
Sep 04 22:44:38.997545 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:38.997505    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:44:40.908402 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:40.908362    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:40.908832 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:40.908417    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:40.922121 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:40.922093    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:41.085445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:41.085413    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:44:45.958732 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:45.958683    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:44:46.059640 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:46.059600    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:44:51.305612 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:51.305559    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:44:59.095343 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.095305    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"]
Sep 04 22:44:59.095883 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.095861    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy"
Sep 04 22:44:59.095955 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.095887    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="001de2e3-d82e-4072-94f3-da88c216f903" containerName="istio-proxy"
Sep 04 22:44:59.099291 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.099271    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.101979 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.101955    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c86c36be-kserve-self-signed-certs\""
Sep 04 22:44:59.109403 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.109379    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"]
Sep 04 22:44:59.132641 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.132610    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:44:59.204382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.204349    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-model-cache\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.204382 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.204397    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tls-certs\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.204600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.204436    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-dshm\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.204600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.204456    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8bbw\" (UniqueName: \"kubernetes.io/projected/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-kube-api-access-b8bbw\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.204600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.204512    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tmp-dir\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.204600 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.204543    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-home\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.305305 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.305260    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tmp-dir\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.305521 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.305319    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-home\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.305521 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.305375    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-model-cache\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.305521 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.305414    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tls-certs\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.305521 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.305452    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-dshm\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.305521 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.305476    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8bbw\" (UniqueName: \"kubernetes.io/projected/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-kube-api-access-b8bbw\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.305781 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.305746    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tmp-dir\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.305838 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.305754    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-home\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.305838 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.305815    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-model-cache\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.307790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.307760    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-dshm\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.307904 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.307881    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tls-certs\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.314841 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.314822    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8bbw\" (UniqueName: \"kubernetes.io/projected/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-kube-api-access-b8bbw\") pod \"e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") " pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.410331 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.410235    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:44:59.684391 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:44:59.684361    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"]
Sep 04 22:44:59.686035 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:44:59.686011    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf97088a4_0e8c_4190_95b6_d36a4a8f80b3.slice/crio-021a0db95d3d30db39bc077c32cd89fab17718bc0fc8da7294cdcf37501fd742 WatchSource:0}: Error finding container 021a0db95d3d30db39bc077c32cd89fab17718bc0fc8da7294cdcf37501fd742: Status 404 returned error can't find the container with id 021a0db95d3d30db39bc077c32cd89fab17718bc0fc8da7294cdcf37501fd742
Sep 04 22:45:00.176315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:00.176272    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj" event={"ID":"f97088a4-0e8c-4190-95b6-d36a4a8f80b3","Type":"ContainerStarted","Data":"6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda"}
Sep 04 22:45:00.176315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:00.176316    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj" event={"ID":"f97088a4-0e8c-4190-95b6-d36a4a8f80b3","Type":"ContainerStarted","Data":"021a0db95d3d30db39bc077c32cd89fab17718bc0fc8da7294cdcf37501fd742"}
Sep 04 22:45:00.199692 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:00.199634    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj" podStartSLOduration=1.199618199 podStartE2EDuration="1.199618199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:44:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:00.198178225 +0000 UTC m=+2183.716900758" watchObservedRunningTime="2026-09-04 22:45:00.199618199 +0000 UTC m=+2183.718340732"
Sep 04 22:45:01.997376 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:01.997337    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:45:09.410540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:09.410496    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:45:09.410996 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:09.410743    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:45:09.423208 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:09.423182    2576 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:45:10.240829 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:10.240799    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:45:18.172104 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:18.172068    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"]
Sep 04 22:45:18.195752 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:18.195705    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:45:19.907558 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:19.907524    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"]
Sep 04 22:45:20.018143 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:20.018098    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:45:20.298641 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:20.298599    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:45:29.496537 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:29.496498    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:45:33.192555 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.192489    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk" podUID="b1dbec85-4bad-4218-a877-3d8126dfacb7" containerName="main" containerID="cri-o://3c9b7bc8e8f901cd4e44b09c493cc44e9def6a949c76c6a2865ca4274646ecf8" gracePeriod=45
Sep 04 22:45:33.346451 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.346417    2576 generic.go:356] "Generic (PLEG): container finished" podID="b1dbec85-4bad-4218-a877-3d8126dfacb7" containerID="3c9b7bc8e8f901cd4e44b09c493cc44e9def6a949c76c6a2865ca4274646ecf8" exitCode=0
Sep 04 22:45:33.346638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.346516    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk" event={"ID":"b1dbec85-4bad-4218-a877-3d8126dfacb7","Type":"ContainerDied","Data":"3c9b7bc8e8f901cd4e44b09c493cc44e9def6a949c76c6a2865ca4274646ecf8"}
Sep 04 22:45:33.442256 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.442229    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:45:33.443367 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.443303    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:45:33.552140 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.552104    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p7v5b\" (UniqueName: \"kubernetes.io/projected/b1dbec85-4bad-4218-a877-3d8126dfacb7-kube-api-access-p7v5b\") pod \"b1dbec85-4bad-4218-a877-3d8126dfacb7\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") "
Sep 04 22:45:33.552140 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.552146    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dbec85-4bad-4218-a877-3d8126dfacb7-tls-certs\") pod \"b1dbec85-4bad-4218-a877-3d8126dfacb7\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") "
Sep 04 22:45:33.552400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.552185    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-tmp-dir\") pod \"b1dbec85-4bad-4218-a877-3d8126dfacb7\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") "
Sep 04 22:45:33.552400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.552214    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-dshm\") pod \"b1dbec85-4bad-4218-a877-3d8126dfacb7\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") "
Sep 04 22:45:33.552400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.552257    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-home\") pod \"b1dbec85-4bad-4218-a877-3d8126dfacb7\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") "
Sep 04 22:45:33.552400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.552287    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-model-cache\") pod \"b1dbec85-4bad-4218-a877-3d8126dfacb7\" (UID: \"b1dbec85-4bad-4218-a877-3d8126dfacb7\") "
Sep 04 22:45:33.552676 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.552528    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b1dbec85-4bad-4218-a877-3d8126dfacb7" (UID: "b1dbec85-4bad-4218-a877-3d8126dfacb7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:45:33.552676 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.552535    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-home" (OuterVolumeSpecName: "home") pod "b1dbec85-4bad-4218-a877-3d8126dfacb7" (UID: "b1dbec85-4bad-4218-a877-3d8126dfacb7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:45:33.552676 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.552613    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-model-cache" (OuterVolumeSpecName: "model-cache") pod "b1dbec85-4bad-4218-a877-3d8126dfacb7" (UID: "b1dbec85-4bad-4218-a877-3d8126dfacb7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:45:33.554384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.554353    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-dshm" (OuterVolumeSpecName: "dshm") pod "b1dbec85-4bad-4218-a877-3d8126dfacb7" (UID: "b1dbec85-4bad-4218-a877-3d8126dfacb7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:45:33.554480 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.554419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1dbec85-4bad-4218-a877-3d8126dfacb7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b1dbec85-4bad-4218-a877-3d8126dfacb7" (UID: "b1dbec85-4bad-4218-a877-3d8126dfacb7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:45:33.554521 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.554482    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1dbec85-4bad-4218-a877-3d8126dfacb7-kube-api-access-p7v5b" (OuterVolumeSpecName: "kube-api-access-p7v5b") pod "b1dbec85-4bad-4218-a877-3d8126dfacb7" (UID: "b1dbec85-4bad-4218-a877-3d8126dfacb7"). InnerVolumeSpecName "kube-api-access-p7v5b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:45:33.653297 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.653258    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:33.653297 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.653290    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:33.653297 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.653299    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:33.653540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.653311    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7v5b\" (UniqueName: \"kubernetes.io/projected/b1dbec85-4bad-4218-a877-3d8126dfacb7-kube-api-access-p7v5b\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:33.653540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.653319    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1dbec85-4bad-4218-a877-3d8126dfacb7-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:33.653540 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:33.653327    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1dbec85-4bad-4218-a877-3d8126dfacb7-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:34.352608 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:34.352579    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"
Sep 04 22:45:34.353113 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:34.352612    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk" event={"ID":"b1dbec85-4bad-4218-a877-3d8126dfacb7","Type":"ContainerDied","Data":"2dc9c391f5895feda17e06b500c1f15d180408c0587cd910f814980222de2620"}
Sep 04 22:45:34.353113 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:34.352664    2576 scope.go:117] "RemoveContainer" containerID="3c9b7bc8e8f901cd4e44b09c493cc44e9def6a949c76c6a2865ca4274646ecf8"
Sep 04 22:45:34.375384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:34.375356    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"]
Sep 04 22:45:34.379458 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:34.379432    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c86c36be-kserve-d5c6f58c9-5n7qk"]
Sep 04 22:45:34.918657 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:34.918616    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1dbec85-4bad-4218-a877-3d8126dfacb7" path="/var/lib/kubelet/pods/b1dbec85-4bad-4218-a877-3d8126dfacb7/volumes"
Sep 04 22:45:34.927362 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:34.927333    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj" podUID="f97088a4-0e8c-4190-95b6-d36a4a8f80b3" containerName="main" containerID="cri-o://6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda" gracePeriod=45
Sep 04 22:45:35.170509 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.170451    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:45:35.359006 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.358965    2576 generic.go:356] "Generic (PLEG): container finished" podID="f97088a4-0e8c-4190-95b6-d36a4a8f80b3" containerID="6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda" exitCode=0
Sep 04 22:45:35.359497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.359018    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"
Sep 04 22:45:35.359497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.359065    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj" event={"ID":"f97088a4-0e8c-4190-95b6-d36a4a8f80b3","Type":"ContainerDied","Data":"6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda"}
Sep 04 22:45:35.359497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.359105    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj" event={"ID":"f97088a4-0e8c-4190-95b6-d36a4a8f80b3","Type":"ContainerDied","Data":"021a0db95d3d30db39bc077c32cd89fab17718bc0fc8da7294cdcf37501fd742"}
Sep 04 22:45:35.359497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.359122    2576 scope.go:117] "RemoveContainer" containerID="6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda"
Sep 04 22:45:35.367705 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.367689    2576 scope.go:117] "RemoveContainer" containerID="6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda"
Sep 04 22:45:35.367947 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:45:35.367923    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda\": container with ID starting with 6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda not found: ID does not exist" containerID="6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda"
Sep 04 22:45:35.368009 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.367957    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda"} err="failed to get container status \"6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda\": rpc error: code = NotFound desc = could not find container \"6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda\": container with ID starting with 6945d391e132bfb76f04a8c9f8d7a4e5c6a461050a7250b8d161b93e3ee1ddda not found: ID does not exist"
Sep 04 22:45:35.369205 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369188    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-model-cache\") pod \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") "
Sep 04 22:45:35.369271 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369218    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-dshm\") pod \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") "
Sep 04 22:45:35.369339 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369318    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tls-certs\") pod \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") "
Sep 04 22:45:35.369450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369357    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b8bbw\" (UniqueName: \"kubernetes.io/projected/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-kube-api-access-b8bbw\") pod \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") "
Sep 04 22:45:35.369450 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369424    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-home\") pod \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") "
Sep 04 22:45:35.369574 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369455    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-model-cache" (OuterVolumeSpecName: "model-cache") pod "f97088a4-0e8c-4190-95b6-d36a4a8f80b3" (UID: "f97088a4-0e8c-4190-95b6-d36a4a8f80b3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:45:35.369574 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369466    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tmp-dir\") pod \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\" (UID: \"f97088a4-0e8c-4190-95b6-d36a4a8f80b3\") "
Sep 04 22:45:35.369681 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369655    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f97088a4-0e8c-4190-95b6-d36a4a8f80b3" (UID: "f97088a4-0e8c-4190-95b6-d36a4a8f80b3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:45:35.369740 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369676    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-home" (OuterVolumeSpecName: "home") pod "f97088a4-0e8c-4190-95b6-d36a4a8f80b3" (UID: "f97088a4-0e8c-4190-95b6-d36a4a8f80b3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:45:35.369846 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369819    2576 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-home\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:35.369941 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369848    2576 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tmp-dir\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:35.369941 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.369861    2576 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-model-cache\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:35.371404 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.371385    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f97088a4-0e8c-4190-95b6-d36a4a8f80b3" (UID: "f97088a4-0e8c-4190-95b6-d36a4a8f80b3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:45:35.371481 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.371398    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-dshm" (OuterVolumeSpecName: "dshm") pod "f97088a4-0e8c-4190-95b6-d36a4a8f80b3" (UID: "f97088a4-0e8c-4190-95b6-d36a4a8f80b3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 04 22:45:35.371571 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.371554    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-kube-api-access-b8bbw" (OuterVolumeSpecName: "kube-api-access-b8bbw") pod "f97088a4-0e8c-4190-95b6-d36a4a8f80b3" (UID: "f97088a4-0e8c-4190-95b6-d36a4a8f80b3"). InnerVolumeSpecName "kube-api-access-b8bbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:45:35.470584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.470551    2576 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-dshm\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:35.470584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.470584    2576 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-tls-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:35.470794 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.470598    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8bbw\" (UniqueName: \"kubernetes.io/projected/f97088a4-0e8c-4190-95b6-d36a4a8f80b3-kube-api-access-b8bbw\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:35.684530 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.684495    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"]
Sep 04 22:45:35.690385 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:35.690356    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c86c36be-kserve-688ff4869f-69mgj"]
Sep 04 22:45:36.919774 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:36.919737    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f97088a4-0e8c-4190-95b6-d36a4a8f80b3" path="/var/lib/kubelet/pods/f97088a4-0e8c-4190-95b6-d36a4a8f80b3/volumes"
Sep 04 22:45:38.802279 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:38.802247    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:45:39.008784 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.008741    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"]
Sep 04 22:45:39.009243 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.009224    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f97088a4-0e8c-4190-95b6-d36a4a8f80b3" containerName="main"
Sep 04 22:45:39.009243 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.009244    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="f97088a4-0e8c-4190-95b6-d36a4a8f80b3" containerName="main"
Sep 04 22:45:39.009422 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.009269    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1dbec85-4bad-4218-a877-3d8126dfacb7" containerName="main"
Sep 04 22:45:39.009422 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.009277    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1dbec85-4bad-4218-a877-3d8126dfacb7" containerName="main"
Sep 04 22:45:39.009422 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.009405    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="f97088a4-0e8c-4190-95b6-d36a4a8f80b3" containerName="main"
Sep 04 22:45:39.009572 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.009423    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1dbec85-4bad-4218-a877-3d8126dfacb7" containerName="main"
Sep 04 22:45:39.013920 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.013892    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.016638 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.016615    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 04 22:45:39.017516 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.017497    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2lmrb\""
Sep 04 22:45:39.017644 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.017546    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 04 22:45:39.017903 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.017886    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 04 22:45:39.017995 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.017976    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 04 22:45:39.023658 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.023628    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"]
Sep 04 22:45:39.097299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.097199    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-metrics-certs\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.097299 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.097251    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-maas-api-tls\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.097494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.097321    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-openshift-service-ca\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.097494 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.097380    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npqhh\" (UniqueName: \"kubernetes.io/projected/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-kube-api-access-npqhh\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.197833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.197791    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-metrics-certs\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.197833 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.197841    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-maas-api-tls\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.198050 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.197877    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-openshift-service-ca\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.198050 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.197903    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npqhh\" (UniqueName: \"kubernetes.io/projected/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-kube-api-access-npqhh\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.198611 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.198584    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-openshift-service-ca\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.200384 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.200361    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-maas-api-tls\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.200501 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.200397    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-metrics-certs\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.206281 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.206262    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npqhh\" (UniqueName: \"kubernetes.io/projected/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-kube-api-access-npqhh\") pod \"maas-api-6d9d77cfb4-zdgww\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") " pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.329341 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.329305    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:39.601473 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.601327    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"]
Sep 04 22:45:39.606995 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:39.606966    2576 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 04 22:45:40.394879 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:40.394837    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww" event={"ID":"a9967b57-cbd9-45b8-9eec-776f4b03e9cf","Type":"ContainerStarted","Data":"25264c74e2d349eb88303f597c69a051a36a22882191b565337715eb445cfb63"}
Sep 04 22:45:41.401322 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:41.401287    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww" event={"ID":"a9967b57-cbd9-45b8-9eec-776f4b03e9cf","Type":"ContainerStarted","Data":"b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6"}
Sep 04 22:45:41.401746 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:41.401346    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:41.421095 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:41.421023    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww" podStartSLOduration=2.28088394 podStartE2EDuration="3.421008808s" totalImagesPullingTime="1.140124868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:41.419638806 +0000 UTC m=+2224.938361339" watchObservedRunningTime="2026-09-04 22:45:41.421008808 +0000 UTC m=+2224.939731341"
Sep 04 22:45:46.176324 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.176288    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"]
Sep 04 22:45:46.176709 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.176460    2576 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww" podUID="a9967b57-cbd9-45b8-9eec-776f4b03e9cf" containerName="maas-api" containerID="cri-o://b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6" gracePeriod=30
Sep 04 22:45:46.182032 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.182005    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:46.211952 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.211915    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"]
Sep 04 22:45:46.215408 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.215393    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.228297 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.228273    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"]
Sep 04 22:45:46.262790 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.262761    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a64135-30b2-47aa-b694-12893e403f66-openshift-service-ca\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.263002 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.262798    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a64135-30b2-47aa-b694-12893e403f66-metrics-certs\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.263002 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.262968    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a64135-30b2-47aa-b694-12893e403f66-maas-api-tls\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.263165 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.263048    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpqss\" (UniqueName: \"kubernetes.io/projected/b5a64135-30b2-47aa-b694-12893e403f66-kube-api-access-vpqss\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.303960 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.303925    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:45:46.363684 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.363654    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a64135-30b2-47aa-b694-12893e403f66-maas-api-tls\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.363837 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.363755    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpqss\" (UniqueName: \"kubernetes.io/projected/b5a64135-30b2-47aa-b694-12893e403f66-kube-api-access-vpqss\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.364009 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.363986    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a64135-30b2-47aa-b694-12893e403f66-openshift-service-ca\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.364133 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.364027    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a64135-30b2-47aa-b694-12893e403f66-metrics-certs\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.365344 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.365315    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b5a64135-30b2-47aa-b694-12893e403f66-openshift-service-ca\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.366828 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.366797    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5a64135-30b2-47aa-b694-12893e403f66-metrics-certs\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.367024 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.367006    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b5a64135-30b2-47aa-b694-12893e403f66-maas-api-tls\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.373639 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.373615    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpqss\" (UniqueName: \"kubernetes.io/projected/b5a64135-30b2-47aa-b694-12893e403f66-kube-api-access-vpqss\") pod \"maas-api-5d9c766c64-vp5t9\" (UID: \"b5a64135-30b2-47aa-b694-12893e403f66\") " pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.427557 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.427503    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:46.428716 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.428690    2576 generic.go:356] "Generic (PLEG): container finished" podID="a9967b57-cbd9-45b8-9eec-776f4b03e9cf" containerID="b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6" exitCode=0
Sep 04 22:45:46.428812 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.428739    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww" event={"ID":"a9967b57-cbd9-45b8-9eec-776f4b03e9cf","Type":"ContainerDied","Data":"b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6"}
Sep 04 22:45:46.428812 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.428785    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww" event={"ID":"a9967b57-cbd9-45b8-9eec-776f4b03e9cf","Type":"ContainerDied","Data":"25264c74e2d349eb88303f597c69a051a36a22882191b565337715eb445cfb63"}
Sep 04 22:45:46.428812 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.428810    2576 scope.go:117] "RemoveContainer" containerID="b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6"
Sep 04 22:45:46.436988 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.436968    2576 scope.go:117] "RemoveContainer" containerID="b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6"
Sep 04 22:45:46.437263 ip-10-0-139-78 kubenswrapper[2576]: E0904 22:45:46.437241    2576 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6\": container with ID starting with b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6 not found: ID does not exist" containerID="b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6"
Sep 04 22:45:46.437315 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.437272    2576 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6"} err="failed to get container status \"b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6\": rpc error: code = NotFound desc = could not find container \"b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6\": container with ID starting with b4173d01c0a7b331096ed4e14e0f69749010318a4b6dcce44df38d7a068e40d6 not found: ID does not exist"
Sep 04 22:45:46.464861 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.464839    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-maas-api-tls\") pod \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") "
Sep 04 22:45:46.464963 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.464871    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npqhh\" (UniqueName: \"kubernetes.io/projected/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-kube-api-access-npqhh\") pod \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") "
Sep 04 22:45:46.464963 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.464921    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-openshift-service-ca\") pod \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") "
Sep 04 22:45:46.465071 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.465013    2576 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-metrics-certs\") pod \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\" (UID: \"a9967b57-cbd9-45b8-9eec-776f4b03e9cf\") "
Sep 04 22:45:46.465395 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.465368    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a9967b57-cbd9-45b8-9eec-776f4b03e9cf" (UID: "a9967b57-cbd9-45b8-9eec-776f4b03e9cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 04 22:45:46.466950 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.466930    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a9967b57-cbd9-45b8-9eec-776f4b03e9cf" (UID: "a9967b57-cbd9-45b8-9eec-776f4b03e9cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:45:46.467116 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.467092    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a9967b57-cbd9-45b8-9eec-776f4b03e9cf" (UID: "a9967b57-cbd9-45b8-9eec-776f4b03e9cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 04 22:45:46.467441 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.467419    2576 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-kube-api-access-npqhh" (OuterVolumeSpecName: "kube-api-access-npqhh") pod "a9967b57-cbd9-45b8-9eec-776f4b03e9cf" (UID: "a9967b57-cbd9-45b8-9eec-776f4b03e9cf"). InnerVolumeSpecName "kube-api-access-npqhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 04 22:45:46.563981 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.563942    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:46.566425 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.566405    2576 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-openshift-service-ca\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:46.566479 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.566427    2576 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-metrics-certs\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:46.566479 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.566436    2576 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-maas-api-tls\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:46.566479 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.566445    2576 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npqhh\" (UniqueName: \"kubernetes.io/projected/a9967b57-cbd9-45b8-9eec-776f4b03e9cf-kube-api-access-npqhh\") on node \"ip-10-0-139-78.ec2.internal\" DevicePath \"\""
Sep 04 22:45:46.836308 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:46.836285    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"]
Sep 04 22:45:46.838956 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:45:46.838929    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5a64135_30b2_47aa_b694_12893e403f66.slice/crio-cbde6c5636b822afca47dcbbb40f231078572b2cbb0e4e9723c69666a2ec52d4 WatchSource:0}: Error finding container cbde6c5636b822afca47dcbbb40f231078572b2cbb0e4e9723c69666a2ec52d4: Status 404 returned error can't find the container with id cbde6c5636b822afca47dcbbb40f231078572b2cbb0e4e9723c69666a2ec52d4
Sep 04 22:45:47.435457 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:47.435428    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9" event={"ID":"b5a64135-30b2-47aa-b694-12893e403f66","Type":"ContainerStarted","Data":"226dd20d7822bfb681dcbb398681468ddf92714a8c830e421b49c965917f7954"}
Sep 04 22:45:47.435821 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:47.435521    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9" event={"ID":"b5a64135-30b2-47aa-b694-12893e403f66","Type":"ContainerStarted","Data":"cbde6c5636b822afca47dcbbb40f231078572b2cbb0e4e9723c69666a2ec52d4"}
Sep 04 22:45:47.435821 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:47.435560    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:47.436655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:47.436621    2576 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"
Sep 04 22:45:47.462602 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:47.462554    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9" podStartSLOduration=0.942313363 podStartE2EDuration="1.462536181s" totalImagesPullingTime="520.222818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:45:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:45:47.462124694 +0000 UTC m=+2230.980847227" watchObservedRunningTime="2026-09-04 22:45:47.462536181 +0000 UTC m=+2230.981258717"
Sep 04 22:45:47.481084 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:47.481049    2576 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"]
Sep 04 22:45:47.485999 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:47.485978    2576 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d77cfb4-zdgww"]
Sep 04 22:45:48.919155 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:48.919123    2576 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9967b57-cbd9-45b8-9eec-776f4b03e9cf" path="/var/lib/kubelet/pods/a9967b57-cbd9-45b8-9eec-776f4b03e9cf/volumes"
Sep 04 22:45:53.391414 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:53.391371    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:45:53.446364 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:53.446337    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d9c766c64-vp5t9"
Sep 04 22:45:59.399610 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:45:59.399526    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:46:05.605108 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:05.605070    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-xwzdn"]
Sep 04 22:46:08.591547 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:08.591512    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b947f4665-x7l5d_3208418d-587d-47c8-889a-1b32a68c57d5/authorino/0.log"
Sep 04 22:46:14.721378 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:14.721347    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cd686f5d7-fh5d7_07170860-e476-40bf-a210-e841ff5297e4/copy-manifests/0.log"
Sep 04 22:46:14.729748 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:14.729715    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cd686f5d7-fh5d7_07170860-e476-40bf-a210-e841ff5297e4/manager/0.log"
Sep 04 22:46:15.304797 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:15.304766    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5d9c766c64-vp5t9_b5a64135-30b2-47aa-b694-12893e403f66/maas-api/0.log"
Sep 04 22:46:15.414135 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:15.414103    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29809350-44nm5_7bd88177-c862-4c42-ab6e-100fc9e68ff0/cleanup/0.log"
Sep 04 22:46:16.422445 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:16.422408    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b947f4665-x7l5d_3208418d-587d-47c8-889a-1b32a68c57d5/authorino/0.log"
Sep 04 22:46:16.658020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:16.657988    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-w4cf5_db110c23-62e9-4e2f-b655-fa093c21079b/manager/0.log"
Sep 04 22:46:17.001914 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:17.001880    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-xwzdn_ed6d1105-257b-414b-897a-569f001d4e02/limitador/0.log"
Sep 04 22:46:17.122413 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:17.122382    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-7nxmd_a9afdd21-4dfc-4204-b80d-6ad317a02a00/manager/0.log"
Sep 04 22:46:17.817944 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:17.817911    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7766956b67-sjbqw_359e04bd-a0b6-4031-bd21-02e24d2452ad/kube-auth-proxy/0.log"
Sep 04 22:46:19.287343 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:19.287311    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-26mrm_2907eae1-38ec-42f0-a8a9-bd1edc73436d/main/0.log"
Sep 04 22:46:24.476070 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.476021    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2c9z6/must-gather-69jwk"]
Sep 04 22:46:24.476505 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.476429    2576 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9967b57-cbd9-45b8-9eec-776f4b03e9cf" containerName="maas-api"
Sep 04 22:46:24.476505 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.476440    2576 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9967b57-cbd9-45b8-9eec-776f4b03e9cf" containerName="maas-api"
Sep 04 22:46:24.476584 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.476509    2576 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9967b57-cbd9-45b8-9eec-776f4b03e9cf" containerName="maas-api"
Sep 04 22:46:24.479946 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.479924    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2c9z6/must-gather-69jwk"
Sep 04 22:46:24.482906 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.482882    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2c9z6\"/\"openshift-service-ca.crt\""
Sep 04 22:46:24.483031 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.482919    2576 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2c9z6\"/\"kube-root-ca.crt\""
Sep 04 22:46:24.483031 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.483008    2576 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2c9z6\"/\"default-dockercfg-6c2zj\""
Sep 04 22:46:24.491271 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.491242    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2c9z6/must-gather-69jwk"]
Sep 04 22:46:24.492581 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.492543    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:46:24.524663 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.524621    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4c2008f-6246-4be1-8293-9adaccc627f9-must-gather-output\") pod \"must-gather-69jwk\" (UID: \"a4c2008f-6246-4be1-8293-9adaccc627f9\") " pod="openshift-must-gather-2c9z6/must-gather-69jwk"
Sep 04 22:46:24.524877 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.524678    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlnhn\" (UniqueName: \"kubernetes.io/projected/a4c2008f-6246-4be1-8293-9adaccc627f9-kube-api-access-wlnhn\") pod \"must-gather-69jwk\" (UID: \"a4c2008f-6246-4be1-8293-9adaccc627f9\") " pod="openshift-must-gather-2c9z6/must-gather-69jwk"
Sep 04 22:46:24.593434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.593388    2576 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log" limit=1024
Sep 04 22:46:24.625871 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.625794    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4c2008f-6246-4be1-8293-9adaccc627f9-must-gather-output\") pod \"must-gather-69jwk\" (UID: \"a4c2008f-6246-4be1-8293-9adaccc627f9\") " pod="openshift-must-gather-2c9z6/must-gather-69jwk"
Sep 04 22:46:24.625871 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.625839    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlnhn\" (UniqueName: \"kubernetes.io/projected/a4c2008f-6246-4be1-8293-9adaccc627f9-kube-api-access-wlnhn\") pod \"must-gather-69jwk\" (UID: \"a4c2008f-6246-4be1-8293-9adaccc627f9\") " pod="openshift-must-gather-2c9z6/must-gather-69jwk"
Sep 04 22:46:24.626185 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.626164    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/a4c2008f-6246-4be1-8293-9adaccc627f9-must-gather-output\") pod \"must-gather-69jwk\" (UID: \"a4c2008f-6246-4be1-8293-9adaccc627f9\") " pod="openshift-must-gather-2c9z6/must-gather-69jwk"
Sep 04 22:46:24.634954 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.634931    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlnhn\" (UniqueName: \"kubernetes.io/projected/a4c2008f-6246-4be1-8293-9adaccc627f9-kube-api-access-wlnhn\") pod \"must-gather-69jwk\" (UID: \"a4c2008f-6246-4be1-8293-9adaccc627f9\") " pod="openshift-must-gather-2c9z6/must-gather-69jwk"
Sep 04 22:46:24.790900 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:24.790865    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2c9z6/must-gather-69jwk"
Sep 04 22:46:25.061539 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:25.061507    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2c9z6/must-gather-69jwk"]
Sep 04 22:46:25.064508 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:46:25.064483    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4c2008f_6246_4be1_8293_9adaccc627f9.slice/crio-e90e90d0c2285a73d3bc9dc656a83ab298548f9622a28ae05d8bc86e50d50770 WatchSource:0}: Error finding container e90e90d0c2285a73d3bc9dc656a83ab298548f9622a28ae05d8bc86e50d50770: Status 404 returned error can't find the container with id e90e90d0c2285a73d3bc9dc656a83ab298548f9622a28ae05d8bc86e50d50770
Sep 04 22:46:25.635984 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:25.635930    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2c9z6/must-gather-69jwk" event={"ID":"a4c2008f-6246-4be1-8293-9adaccc627f9","Type":"ContainerStarted","Data":"e90e90d0c2285a73d3bc9dc656a83ab298548f9622a28ae05d8bc86e50d50770"}
Sep 04 22:46:26.643189 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:26.643147    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2c9z6/must-gather-69jwk" event={"ID":"a4c2008f-6246-4be1-8293-9adaccc627f9","Type":"ContainerStarted","Data":"fdb13c2ac7f942e9a8e385b27c083a59f5ce13b979cafeef41c2e2b5296adb2c"}
Sep 04 22:46:26.643687 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:26.643200    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2c9z6/must-gather-69jwk" event={"ID":"a4c2008f-6246-4be1-8293-9adaccc627f9","Type":"ContainerStarted","Data":"6c182e1bdc36957ec0cadf9363fa28cb62ac3ef4b7de2ab6527c6f5bbb956835"}
Sep 04 22:46:26.661456 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:26.661385    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2c9z6/must-gather-69jwk" podStartSLOduration=1.947606279 podStartE2EDuration="2.661361235s" totalImagesPullingTime="713.754956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-04 22:46:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:46:26.659669025 +0000 UTC m=+2270.178391563" watchObservedRunningTime="2026-09-04 22:46:26.661361235 +0000 UTC m=+2270.180083769"
Sep 04 22:46:27.436412 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:27.436370    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-8hpxw_6dbcb5c1-ac22-4f52-b3b0-fb3c28a5fc9b/global-pull-secret-syncer/0.log"
Sep 04 22:46:27.591243 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:27.591208    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5d6dc_b8c26c57-2178-45fa-8855-ef16d49e2100/konnectivity-agent/0.log"
Sep 04 22:46:27.762156 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:27.762116    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-78.ec2.internal_354628df5e5c0356cd158ffc3ea8a19c/haproxy/0.log"
Sep 04 22:46:32.351163 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:32.348894    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6b947f4665-x7l5d_3208418d-587d-47c8-889a-1b32a68c57d5/authorino/0.log"
Sep 04 22:46:32.440633 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:32.440567    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-w4cf5_db110c23-62e9-4e2f-b655-fa093c21079b/manager/0.log"
Sep 04 22:46:32.543434 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:32.543403    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-xwzdn_ed6d1105-257b-414b-897a-569f001d4e02/limitador/0.log"
Sep 04 22:46:32.642415 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:32.642380    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-6c549b6d85-7nxmd_a9afdd21-4dfc-4204-b80d-6ad317a02a00/manager/0.log"
Sep 04 22:46:34.174423 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.174340    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1fda9de-0d40-446f-bbb8-e52ac79fe2f3/alertmanager/0.log"
Sep 04 22:46:34.201943 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.201907    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1fda9de-0d40-446f-bbb8-e52ac79fe2f3/config-reloader/0.log"
Sep 04 22:46:34.230064 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.230016    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1fda9de-0d40-446f-bbb8-e52ac79fe2f3/kube-rbac-proxy-web/0.log"
Sep 04 22:46:34.293071 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.293015    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1fda9de-0d40-446f-bbb8-e52ac79fe2f3/kube-rbac-proxy/0.log"
Sep 04 22:46:34.323505 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.323472    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1fda9de-0d40-446f-bbb8-e52ac79fe2f3/kube-rbac-proxy-metric/0.log"
Sep 04 22:46:34.359683 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.359648    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1fda9de-0d40-446f-bbb8-e52ac79fe2f3/prom-label-proxy/0.log"
Sep 04 22:46:34.391255 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.391160    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f1fda9de-0d40-446f-bbb8-e52ac79fe2f3/init-config-reloader/0.log"
Sep 04 22:46:34.456731 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.456696    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-6cdcc96965-q4jht_1fbbc3a8-c52a-4592-b802-1b64b5c2a23f/cluster-monitoring-operator/0.log"
Sep 04 22:46:34.586314 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.586268    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7fcccdd6d8-sdvlw_d5bc3346-6eda-44bb-8f0c-7fe984ed48f3/metrics-server/0.log"
Sep 04 22:46:34.614173 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.614142    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-fpj5j_e1b7b90a-1a0b-4350-9b40-73b0781e48f5/monitoring-plugin/0.log"
Sep 04 22:46:34.745590 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.745501    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwnlx_b1f78268-1b3c-4799-9e9c-c30b0a297b6c/node-exporter/0.log"
Sep 04 22:46:34.772188 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.772154    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwnlx_b1f78268-1b3c-4799-9e9c-c30b0a297b6c/kube-rbac-proxy/0.log"
Sep 04 22:46:34.805393 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.805362    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-lwnlx_b1f78268-1b3c-4799-9e9c-c30b0a297b6c/init-textfile/0.log"
Sep 04 22:46:34.943901 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.943856    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5jsh6_7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1/kube-rbac-proxy-main/0.log"
Sep 04 22:46:34.973114 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:34.973079    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5jsh6_7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1/kube-rbac-proxy-self/0.log"
Sep 04 22:46:35.002425 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.002333    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-5jsh6_7a9743d5-dbeb-4b09-ac3a-a83bad6b18c1/openshift-state-metrics/0.log"
Sep 04 22:46:35.049257 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.049208    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70c641ca-9f89-428b-a1c1-057f4b599537/prometheus/0.log"
Sep 04 22:46:35.077598 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.077563    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70c641ca-9f89-428b-a1c1-057f4b599537/config-reloader/0.log"
Sep 04 22:46:35.103615 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.103582    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70c641ca-9f89-428b-a1c1-057f4b599537/thanos-sidecar/0.log"
Sep 04 22:46:35.128319 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.128286    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70c641ca-9f89-428b-a1c1-057f4b599537/kube-rbac-proxy-web/0.log"
Sep 04 22:46:35.157580 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.157556    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70c641ca-9f89-428b-a1c1-057f4b599537/kube-rbac-proxy/0.log"
Sep 04 22:46:35.189269 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.187635    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70c641ca-9f89-428b-a1c1-057f4b599537/kube-rbac-proxy-thanos/0.log"
Sep 04 22:46:35.219020 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.218987    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_70c641ca-9f89-428b-a1c1-057f4b599537/init-config-reloader/0.log"
Sep 04 22:46:35.319131 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.319033    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-78f599db79-962mh_c996da1e-c340-4079-bec0-af1f00551466/prometheus-operator-admission-webhook/0.log"
Sep 04 22:46:35.379962 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.379930    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-87dbd8d9f-25kvp_3bf4d2ce-1024-4505-b2cf-33187bfc6540/telemeter-client/0.log"
Sep 04 22:46:35.424021 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.423992    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-87dbd8d9f-25kvp_3bf4d2ce-1024-4505-b2cf-33187bfc6540/reload/0.log"
Sep 04 22:46:35.453335 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.453285    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-87dbd8d9f-25kvp_3bf4d2ce-1024-4505-b2cf-33187bfc6540/kube-rbac-proxy/0.log"
Sep 04 22:46:35.489168 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.489137    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/thanos-query/0.log"
Sep 04 22:46:35.513955 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.513924    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/kube-rbac-proxy-web/0.log"
Sep 04 22:46:35.539775 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.539739    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/kube-rbac-proxy/0.log"
Sep 04 22:46:35.563796 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.563764    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/prom-label-proxy/0.log"
Sep 04 22:46:35.590400 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.590317    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/kube-rbac-proxy-rules/0.log"
Sep 04 22:46:35.621003 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:35.620978    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-57498c5b8c-4j56l_d92d9363-9172-4535-8eba-c8402302d2b2/kube-rbac-proxy-metrics/0.log"
Sep 04 22:46:36.943030 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:36.942997    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-qt6pg_4e302d63-80ad-4370-8feb-17b2bf538ee2/networking-console-plugin/0.log"
Sep 04 22:46:37.583932 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.583903    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/2.log"
Sep 04 22:46:37.593180 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.593106    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-85b454874b-mj6ng_cb72a802-417c-458b-bca5-c291adeb9bd1/console-operator/3.log"
Sep 04 22:46:37.664894 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.664851    2576 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"]
Sep 04 22:46:37.671253 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.671220    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.678397 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.678361    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"]
Sep 04 22:46:37.810596 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.810548    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-sys\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.810774 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.810617    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-lib-modules\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.810774 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.810736    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-proc\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.810870 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.810775    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bltjt\" (UniqueName: \"kubernetes.io/projected/3451bdca-1477-48d3-8571-a7ab3e7977fc-kube-api-access-bltjt\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.810920 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.810883    2576 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-podres\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.912449 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.912362    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-lib-modules\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.912740 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.912715    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-proc\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.912892 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.912876    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bltjt\" (UniqueName: \"kubernetes.io/projected/3451bdca-1477-48d3-8571-a7ab3e7977fc-kube-api-access-bltjt\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.913078 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.913045    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-podres\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.913230 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.913216    2576 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-sys\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.913562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.913366    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-proc\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.913655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.913446    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-podres\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.913655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.913482    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-sys\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.913655 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.912568    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3451bdca-1477-48d3-8571-a7ab3e7977fc-lib-modules\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.924526 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.924469    2576 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bltjt\" (UniqueName: \"kubernetes.io/projected/3451bdca-1477-48d3-8571-a7ab3e7977fc-kube-api-access-bltjt\") pod \"perf-node-gather-daemonset-txrpg\" (UID: \"3451bdca-1477-48d3-8571-a7ab3e7977fc\") " pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:37.985497 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:37.985454    2576 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:38.290618 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:38.290581    2576 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"]
Sep 04 22:46:38.301289 ip-10-0-139-78 kubenswrapper[2576]: W0904 22:46:38.301251    2576 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3451bdca_1477_48d3_8571_a7ab3e7977fc.slice/crio-4bcb29b2a776252ffc0cc9eeba937148a1625e69b2e2adb462b9957ac511b305 WatchSource:0}: Error finding container 4bcb29b2a776252ffc0cc9eeba937148a1625e69b2e2adb462b9957ac511b305: Status 404 returned error can't find the container with id 4bcb29b2a776252ffc0cc9eeba937148a1625e69b2e2adb462b9957ac511b305
Sep 04 22:46:38.739674 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:38.739631    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg" event={"ID":"3451bdca-1477-48d3-8571-a7ab3e7977fc","Type":"ContainerStarted","Data":"101d49ffd68963bc93793aabbcfdc028b0f9259cccfd8d8bccb03a3fa6e7b059"}
Sep 04 22:46:38.739875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:38.739705    2576 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg" event={"ID":"3451bdca-1477-48d3-8571-a7ab3e7977fc","Type":"ContainerStarted","Data":"4bcb29b2a776252ffc0cc9eeba937148a1625e69b2e2adb462b9957ac511b305"}
Sep 04 22:46:38.739875 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:38.739754    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:38.758068 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:38.757980    2576 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg" podStartSLOduration=1.7579582839999999 podStartE2EDuration="1.757958284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-04 22:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-04 22:46:38.755274316 +0000 UTC m=+2282.273996851" watchObservedRunningTime="2026-09-04 22:46:38.757958284 +0000 UTC m=+2282.276680819"
Sep 04 22:46:40.060432 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:40.060396    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-79rjk_2faabeaa-73ea-4a77-b707-765541109e4d/dns/0.log"
Sep 04 22:46:40.090801 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:40.090766    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-79rjk_2faabeaa-73ea-4a77-b707-765541109e4d/kube-rbac-proxy/0.log"
Sep 04 22:46:40.226948 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:40.226915    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5mt7k_862daba0-07cf-499c-b35f-6a0d34c49cc9/dns-node-resolver/0.log"
Sep 04 22:46:40.922695 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:40.922651    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ftrbj_8add73da-6207-468b-a38f-5db9c6261331/node-ca/0.log"
Sep 04 22:46:42.224200 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:42.224171    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7766956b67-sjbqw_359e04bd-a0b6-4031-bd21-02e24d2452ad/kube-auth-proxy/0.log"
Sep 04 22:46:43.288937 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:43.288906    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-76mz2_f6434544-e7c3-4235-850c-3d4471c2c030/serve-healthcheck-canary/0.log"
Sep 04 22:46:43.885330 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:43.885297    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-58wlk_85c7d188-f68c-4717-884d-fbf00586d73d/insights-operator/0.log"
Sep 04 22:46:43.889739 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:43.889709    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-58wlk_85c7d188-f68c-4717-884d-fbf00586d73d/insights-operator/1.log"
Sep 04 22:46:43.912567 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:43.912537    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5bmbz_4735f7ca-25fc-47b6-9f79-1479438a2a53/kube-rbac-proxy/0.log"
Sep 04 22:46:43.935284 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:43.935248    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5bmbz_4735f7ca-25fc-47b6-9f79-1479438a2a53/exporter/0.log"
Sep 04 22:46:43.960283 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:43.960251    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5bmbz_4735f7ca-25fc-47b6-9f79-1479438a2a53/extractor/0.log"
Sep 04 22:46:44.755562 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:44.755529    2576 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2c9z6/perf-node-gather-daemonset-txrpg"
Sep 04 22:46:46.836011 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:46.835963    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cd686f5d7-fh5d7_07170860-e476-40bf-a210-e841ff5297e4/manager/0.log"
Sep 04 22:46:46.860918 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:46.860890    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cd686f5d7-fh5d7_07170860-e476-40bf-a210-e841ff5297e4/copy-manifests/0.log"
Sep 04 22:46:48.420099 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:48.420065    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-868df85875-xl62t_db41e725-3569-4528-ac27-448b8508b045/manager/0.log"
Sep 04 22:46:48.852461 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:48.852431    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7766956b67-sjbqw_359e04bd-a0b6-4031-bd21-02e24d2452ad/kube-auth-proxy/0.log"
Sep 04 22:46:50.752635 ip-10-0-139-78 kubenswrapper[2576]: I0904 22:46:50.752604    2576 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7766956b67-sjbqw_359e04bd-a0b6-4031-bd21-02e24d2452ad/kube-auth-proxy/0.log"