Sep 17 18:02:42.826279 ip-10-0-142-105 systemd[1]: Starting Kubernetes Kubelet...
Sep 17 18:02:46.423345 ip-10-0-142-105 kubenswrapper[2589]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:02:46.423345 ip-10-0-142-105 kubenswrapper[2589]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 17 18:02:46.423345 ip-10-0-142-105 kubenswrapper[2589]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:02:46.423345 ip-10-0-142-105 kubenswrapper[2589]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 17 18:02:46.423345 ip-10-0-142-105 kubenswrapper[2589]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 17 18:02:46.424057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.423426    2589 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 17 18:02:46.428099 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428082    2589 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:02:46.428099 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428099    2589 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428103    2589 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428107    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428110    2589 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428113    2589 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428116    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428119    2589 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428121    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428125    2589 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428127    2589 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428130    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428133    2589 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428137    2589 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428141    2589 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428144    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428147    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428150    2589 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428152    2589 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428155    2589 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:02:46.428161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428158    2589 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428160    2589 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428163    2589 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428167    2589 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428170    2589 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428173    2589 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428176    2589 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428183    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428186    2589 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428189    2589 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428191    2589 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428194    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428196    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428199    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428201    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428204    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428206    2589 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428209    2589 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428211    2589 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428213    2589 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:02:46.428614 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428216    2589 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428219    2589 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428221    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428223    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428226    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428228    2589 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428231    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428233    2589 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428236    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428238    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428241    2589 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428243    2589 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428246    2589 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428249    2589 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428252    2589 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428256    2589 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428258    2589 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428261    2589 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428263    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428266    2589 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:02:46.429123 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428268    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428271    2589 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428274    2589 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428276    2589 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428279    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428281    2589 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428284    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428286    2589 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428289    2589 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428291    2589 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428295    2589 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428298    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428301    2589 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428303    2589 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428306    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428308    2589 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428311    2589 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428316    2589 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428319    2589 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428321    2589 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:02:46.429622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428324    2589 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428326    2589 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428329    2589 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428333    2589 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428336    2589 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428339    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428342    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428345    2589 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428348    2589 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428350    2589 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428353    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428355    2589 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428358    2589 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428361    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428363    2589 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428366    2589 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428369    2589 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428371    2589 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.428375    2589 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:02:46.430119 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429070    2589 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429078    2589 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429081    2589 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429084    2589 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429087    2589 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429090    2589 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429093    2589 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429095    2589 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429098    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429101    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429103    2589 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429107    2589 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429110    2589 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429113    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429116    2589 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429118    2589 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429121    2589 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429126    2589 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429129    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429133    2589 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:02:46.430615 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429135    2589 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429140    2589 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429144    2589 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429147    2589 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429150    2589 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429153    2589 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429156    2589 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429159    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429161    2589 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429164    2589 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429167    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429170    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429172    2589 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429175    2589 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429178    2589 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429181    2589 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429184    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429186    2589 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429189    2589 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429192    2589 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:02:46.431172 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429194    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429197    2589 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429200    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429202    2589 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429205    2589 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429207    2589 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429210    2589 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429212    2589 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429216    2589 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429222    2589 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429225    2589 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429228    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429233    2589 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429236    2589 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429239    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429242    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429245    2589 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429247    2589 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429250    2589 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:02:46.431666 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429252    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429255    2589 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429257    2589 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429260    2589 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429263    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429265    2589 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429267    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429270    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429273    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429275    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429278    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429280    2589 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429283    2589 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429285    2589 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429288    2589 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429290    2589 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429293    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429295    2589 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429298    2589 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429301    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:02:46.432149 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429304    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429306    2589 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429310    2589 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429313    2589 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429316    2589 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429318    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429321    2589 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429324    2589 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429326    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429329    2589 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429331    2589 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429334    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429336    2589 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429339    2589 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429341    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429344    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429347    2589 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429349    2589 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429352    2589 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429354    2589 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:02:46.432672 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429732    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429738    2589 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429741    2589 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429744    2589 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429747    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429750    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429753    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429756    2589 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429759    2589 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429761    2589 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429765    2589 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429767    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429770    2589 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429773    2589 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429779    2589 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429782    2589 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429785    2589 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429788    2589 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429791    2589 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:02:46.433161 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429793    2589 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429796    2589 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429798    2589 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429801    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429803    2589 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429806    2589 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429808    2589 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429811    2589 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429813    2589 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429816    2589 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429819    2589 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429821    2589 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429824    2589 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429826    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429829    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429832    2589 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429836    2589 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429839    2589 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429841    2589 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:02:46.433616 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429844    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429846    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429849    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429852    2589 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429854    2589 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429857    2589 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429859    2589 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429862    2589 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429867    2589 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429869    2589 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429874    2589 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429876    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429879    2589 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429881    2589 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429884    2589 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429886    2589 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429889    2589 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429891    2589 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429894    2589 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429896    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:02:46.434096 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429898    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429901    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429903    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429905    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429908    2589 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429910    2589 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429913    2589 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429915    2589 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429918    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429920    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429923    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429925    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429927    2589 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429930    2589 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429933    2589 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429935    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429937    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429940    2589 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429943    2589 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:02:46.434588 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429947    2589 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429951    2589 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429954    2589 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429958    2589 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429960    2589 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429963    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429965    2589 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429968    2589 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429970    2589 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429973    2589 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429975    2589 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429978    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429980    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429983    2589 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429985    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429987    2589 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429990    2589 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429992    2589 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429996    2589 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.429999    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:02:46.435066 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430017    2589 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430020    2589 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430098    2589 flags.go:64] FLAG: --address="0.0.0.0"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430143    2589 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430155    2589 flags.go:64] FLAG: --anonymous-auth="true"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430159    2589 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430163    2589 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430166    2589 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430170    2589 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430174    2589 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430178    2589 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430181    2589 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430185    2589 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430190    2589 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430193    2589 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430197    2589 flags.go:64] FLAG: --cgroup-root=""
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430200    2589 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430203    2589 flags.go:64] FLAG: --client-ca-file=""
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430205    2589 flags.go:64] FLAG: --cloud-provider="external"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430208    2589 flags.go:64] FLAG: --cluster-dns="[]"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430213    2589 flags.go:64] FLAG: --cluster-domain=""
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430215    2589 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430218    2589 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430221    2589 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 17 18:02:46.435545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430224    2589 flags.go:64] FLAG: --container-log-max-files="5"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430228    2589 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430231    2589 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430234    2589 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430237    2589 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430240    2589 flags.go:64] FLAG: --contention-profiling="false"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430243    2589 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430246    2589 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430249    2589 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430251    2589 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430256    2589 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430259    2589 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430261    2589 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430264    2589 flags.go:64] FLAG: --enable-load-reader="false"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430267    2589 flags.go:64] FLAG: --enable-server="true"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430270    2589 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430274    2589 flags.go:64] FLAG: --event-burst="100"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430277    2589 flags.go:64] FLAG: --event-qps="50"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430279    2589 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430282    2589 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430285    2589 flags.go:64] FLAG: --eviction-hard=""
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430290    2589 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430293    2589 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430297    2589 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430300    2589 flags.go:64] FLAG: --eviction-soft=""
Sep 17 18:02:46.436490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430303    2589 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430306    2589 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430309    2589 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430312    2589 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430315    2589 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430317    2589 flags.go:64] FLAG: --fail-swap-on="true"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430320    2589 flags.go:64] FLAG: --feature-gates=""
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430323    2589 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430326    2589 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430329    2589 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430332    2589 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430335    2589 flags.go:64] FLAG: --healthz-port="10248"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430338    2589 flags.go:64] FLAG: --help="false"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430341    2589 flags.go:64] FLAG: --hostname-override="ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430344    2589 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430347    2589 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430350    2589 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430353    2589 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430357    2589 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430359    2589 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430362    2589 flags.go:64] FLAG: --image-service-endpoint=""
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430365    2589 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430368    2589 flags.go:64] FLAG: --kube-api-burst="100"
Sep 17 18:02:46.437141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430370    2589 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430374    2589 flags.go:64] FLAG: --kube-api-qps="50"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430377    2589 flags.go:64] FLAG: --kube-reserved=""
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430380    2589 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430382    2589 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430386    2589 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430389    2589 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430393    2589 flags.go:64] FLAG: --lock-file=""
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430396    2589 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430399    2589 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430402    2589 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430407    2589 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430409    2589 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430412    2589 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430415    2589 flags.go:64] FLAG: --logging-format="text"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430418    2589 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430421    2589 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430424    2589 flags.go:64] FLAG: --manifest-url=""
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430427    2589 flags.go:64] FLAG: --manifest-url-header=""
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430431    2589 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430434    2589 flags.go:64] FLAG: --max-open-files="1000000"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430438    2589 flags.go:64] FLAG: --max-pods="110"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430441    2589 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430444    2589 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430446    2589 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 17 18:02:46.437726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430449    2589 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430452    2589 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430455    2589 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430458    2589 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430474    2589 flags.go:64] FLAG: --node-status-max-images="50"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430477    2589 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430480    2589 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430483    2589 flags.go:64] FLAG: --pod-cidr=""
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430485    2589 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430491    2589 flags.go:64] FLAG: --pod-manifest-path=""
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430494    2589 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430497    2589 flags.go:64] FLAG: --pods-per-core="0"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430500    2589 flags.go:64] FLAG: --port="10250"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430503    2589 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430508    2589 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-00a559f6f4462c63e"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430511    2589 flags.go:64] FLAG: --qos-reserved=""
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430514    2589 flags.go:64] FLAG: --read-only-port="10255"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430516    2589 flags.go:64] FLAG: --register-node="true"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430519    2589 flags.go:64] FLAG: --register-with-taints=""
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430523    2589 flags.go:64] FLAG: --registry-burst="10"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430526    2589 flags.go:64] FLAG: --registry-qps="5"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430528    2589 flags.go:64] FLAG: --reserved-cpus=""
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430531    2589 flags.go:64] FLAG: --reserved-memory=""
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430534    2589 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430537    2589 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 17 18:02:46.438367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430541    2589 flags.go:64] FLAG: --rotate-certificates="false"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430543    2589 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430546    2589 flags.go:64] FLAG: --runonce="false"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430549    2589 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430552    2589 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430555    2589 flags.go:64] FLAG: --seccomp-default="false"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430558    2589 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430560    2589 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430563    2589 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430566    2589 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430569    2589 flags.go:64] FLAG: --storage-driver-password="root"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430572    2589 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430575    2589 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430578    2589 flags.go:64] FLAG: --storage-driver-user="root"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430580    2589 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430583    2589 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430586    2589 flags.go:64] FLAG: --system-cgroups=""
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430589    2589 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430596    2589 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430599    2589 flags.go:64] FLAG: --tls-cert-file=""
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430602    2589 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430607    2589 flags.go:64] FLAG: --tls-min-version=""
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430610    2589 flags.go:64] FLAG: --tls-private-key-file=""
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430613    2589 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430616    2589 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 17 18:02:46.438992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430619    2589 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430622    2589 flags.go:64] FLAG: --v="2"
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430626    2589 flags.go:64] FLAG: --version="false"
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430630    2589 flags.go:64] FLAG: --vmodule=""
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430633    2589 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.430636    2589 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430763    2589 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430767    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430770    2589 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430773    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430776    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430779    2589 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430782    2589 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430784    2589 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430787    2589 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430790    2589 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430792    2589 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430795    2589 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430797    2589 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430800    2589 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430803    2589 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:02:46.439654 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430805    2589 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430808    2589 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430810    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430813    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430817    2589 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430820    2589 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430823    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430826    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430829    2589 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430832    2589 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430834    2589 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430837    2589 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430840    2589 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430842    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430845    2589 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430848    2589 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430850    2589 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430853    2589 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430855    2589 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430858    2589 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:02:46.440233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430861    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430863    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430866    2589 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430868    2589 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430871    2589 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430874    2589 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430876    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430879    2589 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430882    2589 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430885    2589 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430887    2589 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430890    2589 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430893    2589 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430896    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430907    2589 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430910    2589 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430913    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430916    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430918    2589 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430922    2589 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:02:46.440739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430924    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430928    2589 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430931    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430934    2589 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430936    2589 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430939    2589 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430941    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430944    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430946    2589 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430949    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430952    2589 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430954    2589 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430957    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430959    2589 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430962    2589 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430964    2589 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430969    2589 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430972    2589 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430975    2589 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430978    2589 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:02:46.441269 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430981    2589 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430983    2589 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430986    2589 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430989    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430992    2589 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430995    2589 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.430997    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431014    2589 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431019    2589 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431022    2589 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431025    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431028    2589 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431031    2589 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431034    2589 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431037    2589 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431040    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431044    2589 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431048    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431051    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:02:46.441822 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431054    2589 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431056    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431059    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431061    2589 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.431064    2589 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.431069    2589 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.438342    2589 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.438358    2589 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.438377    2589 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.438386    2589 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438441    2589 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438446    2589 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438449    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438453    2589 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:02:46.442320 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438456    2589 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438459    2589 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438462    2589 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438475    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438479    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438482    2589 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438485    2589 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438487    2589 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438490    2589 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438494    2589 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438498    2589 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438502    2589 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438506    2589 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438511    2589 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438515    2589 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438518    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438521    2589 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438524    2589 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:02:46.442691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438527    2589 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438530    2589 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438533    2589 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438535    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438539    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438541    2589 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438544    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438548    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438550    2589 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438553    2589 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438556    2589 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438558    2589 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438561    2589 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438563    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438566    2589 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438568    2589 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438571    2589 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438573    2589 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438576    2589 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:02:46.443164 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438578    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438581    2589 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438583    2589 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438586    2589 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438588    2589 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438591    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438593    2589 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438596    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438599    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438601    2589 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438604    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438607    2589 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438609    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438611    2589 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438614    2589 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438616    2589 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438619    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438621    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438624    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438626    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:02:46.443636 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438629    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438632    2589 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438634    2589 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438637    2589 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438640    2589 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438643    2589 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438645    2589 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438648    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438650    2589 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438653    2589 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438655    2589 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438658    2589 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438660    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438663    2589 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438666    2589 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438668    2589 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438670    2589 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438673    2589 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438676    2589 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438678    2589 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:02:46.444150 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438681    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438683    2589 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438685    2589 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438688    2589 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438690    2589 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438692    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438695    2589 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438698    2589 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438700    2589 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438702    2589 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438705    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438707    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438710    2589 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438712    2589 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438715    2589 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438718    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438721    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:02:46.444671 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438724    2589 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.438729    2589 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438841    2589 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438847    2589 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438850    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438853    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438856    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438859    2589 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438861    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438864    2589 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438867    2589 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438870    2589 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438873    2589 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438875    2589 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 17 18:02:46.445159 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438878    2589 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438881    2589 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438884    2589 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438886    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438889    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438892    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438895    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438897    2589 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438900    2589 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438902    2589 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438905    2589 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438907    2589 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438910    2589 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438912    2589 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438915    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438918    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438920    2589 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438923    2589 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438925    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438930    2589 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 17 18:02:46.445506 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438933    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438936    2589 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438938    2589 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438941    2589 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438943    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438946    2589 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438950    2589 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438953    2589 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438956    2589 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438959    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438961    2589 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438964    2589 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438966    2589 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438969    2589 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438971    2589 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438974    2589 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438976    2589 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438978    2589 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438981    2589 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 17 18:02:46.446028 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438983    2589 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438985    2589 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438988    2589 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438990    2589 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438993    2589 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438996    2589 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.438998    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439014    2589 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439017    2589 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439020    2589 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439023    2589 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439026    2589 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439029    2589 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439032    2589 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439034    2589 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439037    2589 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439040    2589 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439042    2589 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439045    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 17 18:02:46.446512 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439047    2589 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439049    2589 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439052    2589 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439054    2589 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439056    2589 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439059    2589 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439061    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439064    2589 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439066    2589 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439069    2589 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439071    2589 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439074    2589 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439076    2589 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439079    2589 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439081    2589 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439084    2589 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439087    2589 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439089    2589 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439091    2589 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439094    2589 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 17 18:02:46.447089 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439097    2589 feature_gate.go:328] unrecognized feature gate: Example2
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439100    2589 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439102    2589 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439105    2589 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439109    2589 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439111    2589 feature_gate.go:328] unrecognized feature gate: Example
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439114    2589 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439116    2589 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:46.439119    2589 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.439123    2589 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.439223    2589 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.441220    2589 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.442306    2589 server.go:1019] "Starting client certificate rotation"
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.442399    2589 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:02:46.447906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.442431    2589 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 17 18:02:46.448366 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.444543    2589 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:02:46.448366 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.446847    2589 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 17 18:02:46.454978 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.454957    2589 log.go:25] "Validated CRI v1 runtime API"
Sep 17 18:02:46.458337 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.458311    2589 log.go:25] "Validated CRI v1 image API"
Sep 17 18:02:46.459663 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.459649    2589 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 17 18:02:46.462054 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.462036    2589 fs.go:135] Filesystem UUIDs: map[46474b7f-8fbd-41bb-aa6d-d0dbaf022bae:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 909f703d-e3d2-4080-b8dc-b849f994a8f7:/dev/nvme0n1p3]
Sep 17 18:02:46.462106 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.462053    2589 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 17 18:02:46.466856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.466749    2589 manager.go:216] Machine: {Timestamp:2026-09-17 18:02:46.46622719 +0000 UTC m=+2.145913035 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3083174 MemoryCapacity:32812154880 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec29bc31523dd71c6a0210b3f56ed78c SystemUUID:ec29bc31-523d-d71c-6a02-10b3f56ed78c BootID:b7863a32-33e4-4af9-9ab5-7028cd87eb9d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406077440 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f1:c9:5c:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f1:c9:5c:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:7a:33:7c:33:20:31 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812154880 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 17 18:02:46.466856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.466855    2589 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 17 18:02:46.466955 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.466923    2589 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 17 18:02:46.469690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.469668    2589 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 17 18:02:46.469825 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.469692    2589 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-142-105.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 17 18:02:46.470941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.470931    2589 topology_manager.go:138] "Creating topology manager with none policy"
Sep 17 18:02:46.470974 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.470944    2589 container_manager_linux.go:309] "Creating device plugin manager"
Sep 17 18:02:46.470974 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.470952    2589 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:02:46.470974 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.470962    2589 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 17 18:02:46.471075 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.471045    2589 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 17 18:02:46.475086 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.475075    2589 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:02:46.475229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.475220    2589 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 17 18:02:46.475267 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.475255    2589 kubelet.go:486] "Attempting to sync node with API server"
Sep 17 18:02:46.475267 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.475265    2589 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 17 18:02:46.475333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.475275    2589 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 17 18:02:46.475333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.475284    2589 kubelet.go:398] "Adding apiserver pod source"
Sep 17 18:02:46.475333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.475295    2589 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 17 18:02:46.475633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.475621    2589 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:02:46.475668 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.475640    2589 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 17 18:02:46.477914 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.477898    2589 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 17 18:02:46.480082 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.480066    2589 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 17 18:02:46.480140 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.480097    2589 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 17 18:02:46.481218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481203    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 17 18:02:46.481264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481230    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 17 18:02:46.481264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481237    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 17 18:02:46.481264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481242    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 17 18:02:46.481264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481250    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 17 18:02:46.481264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481255    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 17 18:02:46.481400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481269    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 17 18:02:46.481400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481274    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 17 18:02:46.481400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481288    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 17 18:02:46.481400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481293    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 17 18:02:46.481400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481305    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 17 18:02:46.481400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481310    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 17 18:02:46.481400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481335    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 17 18:02:46.481400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.481343    2589 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 17 18:02:46.483889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.483868    2589 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:02:46.485222 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.485209    2589 server.go:1268] "Started kubelet"
Sep 17 18:02:46.485287 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.485263    2589 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 17 18:02:46.485373 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.485342    2589 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 17 18:02:46.485483 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.485385    2589 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 17 18:02:46.519982 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.519947    2589 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 17 18:02:46.520203 ip-10-0-142-105 systemd[1]: Started Kubernetes Kubelet.
Sep 17 18:02:46.522148 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.522114    2589 server.go:310] "Adding debug handlers to kubelet server"
Sep 17 18:02:46.522256 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.522236    2589 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 17 18:02:46.523356 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.523337    2589 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-142-105.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 17 18:02:46.523446 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.523379    2589 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 17 18:02:46.524126 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.524102    2589 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-142-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 17 18:02:46.527839 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.527827    2589 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 17 18:02:46.527898 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.527845    2589 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 17 18:02:46.528067 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.527144    2589 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-105.ec2.internal.18d62db7212c81c3  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-105.ec2.internal,UID:ip-10-0-142-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-142-105.ec2.internal,},FirstTimestamp:2026-09-17 18:02:46.485180867 +0000 UTC m=+2.164866743,LastTimestamp:2026-09-17 18:02:46.485180867 +0000 UTC m=+2.164866743,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-105.ec2.internal,}"
Sep 17 18:02:46.528067 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.528053    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:46.528353 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.528339    2589 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 17 18:02:46.528418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.528341    2589 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 17 18:02:46.528418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.528368    2589 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 17 18:02:46.528546 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.528533    2589 reconstruct.go:97] "Volume reconstruction finished"
Sep 17 18:02:46.528596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.528548    2589 reconciler.go:29] "Reconciler: start to sync state"
Sep 17 18:02:46.528834 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.528732    2589 factory.go:55] Registering systemd factory
Sep 17 18:02:46.528834 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.528756    2589 factory.go:223] Registration of the systemd container factory successfully
Sep 17 18:02:46.528999 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.528977    2589 factory.go:153] Registering CRI-O factory
Sep 17 18:02:46.528999 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.528989    2589 factory.go:223] Registration of the crio container factory successfully
Sep 17 18:02:46.529119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.529067    2589 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 17 18:02:46.529119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.529088    2589 factory.go:103] Registering Raw factory
Sep 17 18:02:46.529119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.529103    2589 manager.go:1196] Started watching for new ooms in manager
Sep 17 18:02:46.529488 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.529474    2589 manager.go:349] Starting recovery of all containers
Sep 17 18:02:46.536760 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.536741    2589 manager.go:354] Recovery completed
Sep 17 18:02:46.538855 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.538797    2589 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-142-105.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 17 18:02:46.538997 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.538907    2589 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 17 18:02:46.541593 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.541581    2589 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:02:46.543285 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.543251    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:02:46.543350 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.543299    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:02:46.543350 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.543311    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:02:46.543765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.543749    2589 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 17 18:02:46.543765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.543762    2589 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 17 18:02:46.543889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.543776    2589 state_mem.go:36] "Initialized new in-memory state store"
Sep 17 18:02:46.543889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.543775    2589 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:02:46.544470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.544457    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:02:46.544557 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.544488    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:02:46.544557 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.544503    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:02:46.545077 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.545063    2589 policy_none.go:49] "None policy: Start"
Sep 17 18:02:46.545138 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.545082    2589 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 17 18:02:46.545138 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.545096    2589 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 17 18:02:46.545729 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.545658    2589 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-105.ec2.internal.18d62db724a31a45  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-105.ec2.internal,UID:ip-10-0-142-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-105.ec2.internal,},FirstTimestamp:2026-09-17 18:02:46.543284805 +0000 UTC m=+2.222970653,LastTimestamp:2026-09-17 18:02:46.543284805 +0000 UTC m=+2.222970653,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-105.ec2.internal,}"
Sep 17 18:02:46.546224 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.546212    2589 policy_none.go:47] "Start"
Sep 17 18:02:46.559224 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.559134    2589 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-105.ec2.internal.18d62db724a36502  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-105.ec2.internal,UID:ip-10-0-142-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-105.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-105.ec2.internal,},FirstTimestamp:2026-09-17 18:02:46.543303938 +0000 UTC m=+2.222989782,LastTimestamp:2026-09-17 18:02:46.543303938 +0000 UTC m=+2.222989782,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-105.ec2.internal,}"
Sep 17 18:02:46.571163 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.570977    2589 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-105.ec2.internal.18d62db724a39088  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-105.ec2.internal,UID:ip-10-0-142-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-142-105.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-142-105.ec2.internal,},FirstTimestamp:2026-09-17 18:02:46.54331508 +0000 UTC m=+2.223000925,LastTimestamp:2026-09-17 18:02:46.54331508 +0000 UTC m=+2.223000925,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-105.ec2.internal,}"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.575211    2589 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8btq6"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.581308    2589 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-105.ec2.internal.18d62db724a31a45\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-105.ec2.internal.18d62db724a31a45  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-105.ec2.internal,UID:ip-10-0-142-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-142-105.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-142-105.ec2.internal,},FirstTimestamp:2026-09-17 18:02:46.543284805 +0000 UTC m=+2.222970653,LastTimestamp:2026-09-17 18:02:46.544471787 +0000 UTC m=+2.224157641,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-105.ec2.internal,}"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.585316    2589 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-8btq6"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.589596    2589 manager.go:337] "Starting Device Plugin manager"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.589635    2589 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.589648    2589 server.go:87] "Starting device plugin registration server"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.589946    2589 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.589964    2589 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.590088    2589 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.590189    2589 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.590198    2589 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.590309    2589 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-142-105.ec2.internal.18d62db724a36502\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-142-105.ec2.internal.18d62db724a36502  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-142-105.ec2.internal,UID:ip-10-0-142-105.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-142-105.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-142-105.ec2.internal,},FirstTimestamp:2026-09-17 18:02:46.543303938 +0000 UTC m=+2.222989782,LastTimestamp:2026-09-17 18:02:46.544494573 +0000 UTC m=+2.224180419,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-142-105.ec2.internal,}"
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.590754    2589 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 17 18:02:46.593315 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.590842    2589 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:46.675737 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.675656    2589 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 17 18:02:46.676938 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.676915    2589 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 17 18:02:46.677033 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.676969    2589 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 17 18:02:46.677033 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.676988    2589 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 17 18:02:46.677117 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.677038    2589 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 17 18:02:46.679671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.679654    2589 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:02:46.690061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.690036    2589 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:02:46.690843 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.690825    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:02:46.690922 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.690860    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:02:46.690922 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.690875    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:02:46.690922 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.690903    2589 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.700727 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.700713    2589 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.700796 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.700732    2589 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-142-105.ec2.internal\": node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:46.765141 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.765110    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:46.777708 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.777686    2589 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"]
Sep 17 18:02:46.777785 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.777777    2589 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:02:46.778631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.778615    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:02:46.778700 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.778642    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:02:46.778700 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.778652    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:02:46.780395 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.780382    2589 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:02:46.780530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.780516    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.780588 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.780552    2589 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:02:46.781083 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.781065    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:02:46.781169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.781081    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:02:46.781169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.781094    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:02:46.781169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.781106    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:02:46.781169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.781118    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:02:46.781169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.781109    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:02:46.782702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.782684    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.782759 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.782720    2589 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:02:46.783374 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.783359    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:02:46.783439 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.783390    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:02:46.783439 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.783402    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:02:46.805629 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.805607    2589 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-105.ec2.internal\" not found" node="ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.809736 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.809721    2589 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-142-105.ec2.internal\" not found" node="ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.865935 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.865917    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:46.929722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.929649    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5f4cf6b5e1eea622d2a62af388775eb4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"5f4cf6b5e1eea622d2a62af388775eb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.929722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.929678    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f4cf6b5e1eea622d2a62af388775eb4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"5f4cf6b5e1eea622d2a62af388775eb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.929722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:46.929695    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ff6991f08a902cdf413530d4bdd103e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" (UID: \"3ff6991f08a902cdf413530d4bdd103e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:46.966015 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:46.965991    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.030844 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.030819    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5f4cf6b5e1eea622d2a62af388775eb4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"5f4cf6b5e1eea622d2a62af388775eb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:47.030927 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.030856    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f4cf6b5e1eea622d2a62af388775eb4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"5f4cf6b5e1eea622d2a62af388775eb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:47.030927 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.030879    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ff6991f08a902cdf413530d4bdd103e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" (UID: \"3ff6991f08a902cdf413530d4bdd103e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:47.030927 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.030888    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5f4cf6b5e1eea622d2a62af388775eb4-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"5f4cf6b5e1eea622d2a62af388775eb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:47.031049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.030922    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f4cf6b5e1eea622d2a62af388775eb4-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal\" (UID: \"5f4cf6b5e1eea622d2a62af388775eb4\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:47.031049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.030931    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3ff6991f08a902cdf413530d4bdd103e-config\") pod \"kube-apiserver-proxy-ip-10-0-142-105.ec2.internal\" (UID: \"3ff6991f08a902cdf413530d4bdd103e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:47.066574 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:47.066555    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.109744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.109712    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:47.113330 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.113314    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:47.167102 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:47.167078    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.267605 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:47.267583    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.368055 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:47.368021    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.441547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.441511    2589 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 17 18:02:47.441925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.441680    2589 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 17 18:02:47.468798 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:47.468776    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.569468 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:47.569406    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.590591 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.590557    2589 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-16 17:57:46 +0000 UTC" deadline="2028-04-11 04:53:07.557581207 +0000 UTC"
Sep 17 18:02:47.590591 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.590585    2589 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13714h50m19.967000494s"
Sep 17 18:02:47.670040 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:47.670017    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.713100 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.713069    2589 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 17 18:02:47.714093 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.714072    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientMemory"
Sep 17 18:02:47.714188 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.714109    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasNoDiskPressure"
Sep 17 18:02:47.714188 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.714120    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeHasSufficientPID"
Sep 17 18:02:47.771023 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:47.770984    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.800893 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.800866    2589 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:02:47.871492 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:47.871468    2589 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-142-105.ec2.internal\" not found"
Sep 17 18:02:47.933247 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.933214    2589 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:02:47.984215 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:47.984179    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff6991f08a902cdf413530d4bdd103e.slice/crio-1cba8ceb0df0ee684ebc31dea4f15c36159d0219616ba44484bf2e666e376a77 WatchSource:0}: Error finding container 1cba8ceb0df0ee684ebc31dea4f15c36159d0219616ba44484bf2e666e376a77: Status 404 returned error can't find the container with id 1cba8ceb0df0ee684ebc31dea4f15c36159d0219616ba44484bf2e666e376a77
Sep 17 18:02:47.991902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:47.991885    2589 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:02:48.028655 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.028626    2589 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:48.045065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.045047    2589 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:02:48.047684 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.047671    2589 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal"
Sep 17 18:02:48.056500 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.056484    2589 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 17 18:02:48.069636 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.069618    2589 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:02:48.476754 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.476721    2589 apiserver.go:52] "Watching apiserver"
Sep 17 18:02:48.484285 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.484261    2589 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 17 18:02:48.484638 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.484617    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-rb4ph","openshift-multus/multus-psqh9","openshift-network-operator/iptables-alerter-nk555","openshift-ovn-kubernetes/ovnkube-node-m4trh","kube-system/global-pull-secret-syncer-89zbw","kube-system/konnectivity-agent-6bf5d","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q","openshift-cluster-node-tuning-operator/tuned-f2nkl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal","openshift-multus/multus-additional-cni-plugins-t92dt","openshift-multus/network-metrics-daemon-jmfh6","openshift-network-diagnostics/network-check-target-jccnv","kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal","openshift-dns/node-resolver-xlknw"]
Sep 17 18:02:48.487075 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.487058    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.488196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.488180    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.489290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.489276    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.489594 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.489572    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 17 18:02:48.489594 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.489586    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 17 18:02:48.489741 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.489725    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 17 18:02:48.489741 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.489738    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mnbk2\""
Sep 17 18:02:48.490525 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.490503    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 17 18:02:48.490600 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.490524    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 17 18:02:48.490600 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.490539    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 17 18:02:48.490788 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.490771    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.490846 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.490791    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-r5f56\""
Sep 17 18:02:48.491904 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.491885    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 17 18:02:48.492000 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.491933    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wpp9z\""
Sep 17 18:02:48.492000 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.491975    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 17 18:02:48.492240 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.492218    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.493502 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.493485    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:48.493593 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.493551    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:02:48.494356 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.494339    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 17 18:02:48.494854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.494839    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:48.494942 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.494899    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:02:48.495195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495175    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 17 18:02:48.495273 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495201    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 17 18:02:48.495273 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495180    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 17 18:02:48.495273 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495250    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 17 18:02:48.495391 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495321    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-lk6pt\""
Sep 17 18:02:48.495391 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495322    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 17 18:02:48.495471 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495457    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 17 18:02:48.495542 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495527    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:02:48.495594 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495567    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-9stnd\""
Sep 17 18:02:48.495975 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.495961    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.497240 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.497225    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.498391 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.498372    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 17 18:02:48.498888 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.498869    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:02:48.499213 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.499068    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 17 18:02:48.499213 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.499095    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 17 18:02:48.499481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.499464    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 17 18:02:48.499548 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.499467    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-k4c76\""
Sep 17 18:02:48.499924 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.499904    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2nzwr\""
Sep 17 18:02:48.500139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.500122    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 17 18:02:48.500206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.500137    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 17 18:02:48.500697 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.500677    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:48.501025 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.500983    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:02:48.501442 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.501423    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-9tgtr\""
Sep 17 18:02:48.501530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.501453    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 17 18:02:48.501530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.501446    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 17 18:02:48.502448 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.502428    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.505885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.505530    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ccbh2\""
Sep 17 18:02:48.505885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.505615    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 17 18:02:48.505885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.505714    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 17 18:02:48.505885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.505847    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:02:48.529060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.529041    2589 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 17 18:02:48.530118 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.530102    2589 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 17 18:02:48.536595 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536577    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.536685 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536605    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-multus-socket-dir-parent\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.536685 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536629    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-var-lib-cni-bin\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.536685 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536662    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-var-lib-kubelet\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.536812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536685    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dk4mv\" (UniqueName: \"kubernetes.io/projected/77c15d1f-6313-427c-b664-cf17e671cb29-kube-api-access-dk4mv\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.536812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536706    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-dbus\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:48.536812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536730    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-systemd-units\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.536812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536764    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-slash\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.536812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536785    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c909fa0f-361f-4896-9cf5-9b20f2e8ec94-konnectivity-ca\") pod \"konnectivity-agent-6bf5d\" (UID: \"c909fa0f-361f-4896-9cf5-9b20f2e8ec94\") " pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:02:48.536812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536803    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40eaaa5e-fe21-44f7-a769-2778ab800d3f-tmp-dir\") pod \"node-resolver-xlknw\" (UID: \"40eaaa5e-fe21-44f7-a769-2778ab800d3f\") " pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536827    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-var-lib-cni-multus\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536845    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-cni-binary-copy\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536859    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-sysctl-d\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536871    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-host\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536888    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-multus-cni-dir\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536907    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-run-multus-certs\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536922    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-var-lib-kubelet\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536935    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536951    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-multus-conf-dir\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.536982    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-cni-netd\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537024    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-ovnkube-config\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.537062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537048    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-sysconfig\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537071    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77c15d1f-6313-427c-b664-cf17e671cb29-multus-daemon-config\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537093    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537116    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-kubelet\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537140    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-var-lib-openvswitch\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537175    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-tuned\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537204    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-log-socket\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537239    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-cni-bin\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537264    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537282    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccgp6\" (UniqueName: \"kubernetes.io/projected/3d310680-2035-4659-839c-2a1357a32aba-kube-api-access-ccgp6\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537297    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-hostroot\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537311    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5dlw\" (UniqueName: \"kubernetes.io/projected/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-kube-api-access-k5dlw\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537331    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkxf9\" (UniqueName: \"kubernetes.io/projected/973e536d-c06f-4512-b968-10b4cc35589c-kube-api-access-rkxf9\") pod \"iptables-alerter-nk555\" (UID: \"973e536d-c06f-4512-b968-10b4cc35589c\") " pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537364    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-run-netns\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537388    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537418    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjmlf\" (UniqueName: \"kubernetes.io/projected/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-kube-api-access-mjmlf\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.537515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537437    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-systemd\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537451    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-run\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537468    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-sys\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537490    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fjgx\" (UniqueName: \"kubernetes.io/projected/c839ad3e-92e9-4bcd-8a49-495ad550682a-kube-api-access-5fjgx\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537505    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537519    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-run-k8s-cni-cncf-io\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537538    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-node-log\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537574    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-lib-modules\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537589    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-device-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537608    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/973e536d-c06f-4512-b968-10b4cc35589c-host-slash\") pod \"iptables-alerter-nk555\" (UID: \"973e536d-c06f-4512-b968-10b4cc35589c\") " pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537635    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-run-openvswitch\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537678    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-ovn-node-metrics-cert\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537719    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77c15d1f-6313-427c-b664-cf17e671cb29-cni-binary-copy\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537746    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-kubelet-config\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537770    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-env-overrides\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537793    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-modprobe-d\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.538258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537811    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-kubernetes\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537823    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-sysctl-conf\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537840    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-run-ovn\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537885    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-registration-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537916    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs4cg\" (UniqueName: \"kubernetes.io/projected/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-kube-api-access-xs4cg\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537948    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/973e536d-c06f-4512-b968-10b4cc35589c-iptables-alerter-script\") pod \"iptables-alerter-nk555\" (UID: \"973e536d-c06f-4512-b968-10b4cc35589c\") " pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537970    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e8325d4-12ab-4d44-9353-9853c42337ae-host\") pod \"node-ca-rb4ph\" (UID: \"2e8325d4-12ab-4d44-9353-9853c42337ae\") " pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.537995    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e8325d4-12ab-4d44-9353-9853c42337ae-serviceca\") pod \"node-ca-rb4ph\" (UID: \"2e8325d4-12ab-4d44-9353-9853c42337ae\") " pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538035    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-run-netns\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538056    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c839ad3e-92e9-4bcd-8a49-495ad550682a-tmp\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538087    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c909fa0f-361f-4896-9cf5-9b20f2e8ec94-agent-certs\") pod \"konnectivity-agent-6bf5d\" (UID: \"c909fa0f-361f-4896-9cf5-9b20f2e8ec94\") " pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538117    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jk8gm\" (UniqueName: \"kubernetes.io/projected/40eaaa5e-fe21-44f7-a769-2778ab800d3f-kube-api-access-jk8gm\") pod \"node-resolver-xlknw\" (UID: \"40eaaa5e-fe21-44f7-a769-2778ab800d3f\") " pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538140    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-system-cni-dir\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538182    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-run-systemd\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538221    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-etc-openvswitch\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538245    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.538756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538270    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-system-cni-dir\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538288    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-cnibin\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538317    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-cnibin\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538339    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538354    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zws85\" (UniqueName: \"kubernetes.io/projected/2e8325d4-12ab-4d44-9353-9853c42337ae-kube-api-access-zws85\") pod \"node-ca-rb4ph\" (UID: \"2e8325d4-12ab-4d44-9353-9853c42337ae\") " pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538367    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-sys-fs\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538381    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40eaaa5e-fe21-44f7-a769-2778ab800d3f-hosts-file\") pod \"node-resolver-xlknw\" (UID: \"40eaaa5e-fe21-44f7-a769-2778ab800d3f\") " pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538394    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-etc-kubernetes\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538409    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538431    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-ovnkube-script-lib\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538445    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:48.539331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.538458    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-socket-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.539684 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.539438    2589 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 17 18:02:48.562470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.562442    2589 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b85rl"
Sep 17 18:02:48.571740 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.571719    2589 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b85rl"
Sep 17 18:02:48.639514 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639488    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-run-ovn\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.639667 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639527    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-registration-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.639667 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639573    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs4cg\" (UniqueName: \"kubernetes.io/projected/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-kube-api-access-xs4cg\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.639667 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639590    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/973e536d-c06f-4512-b968-10b4cc35589c-iptables-alerter-script\") pod \"iptables-alerter-nk555\" (UID: \"973e536d-c06f-4512-b968-10b4cc35589c\") " pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.639667 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639615    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-run-ovn\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.639667 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639613    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e8325d4-12ab-4d44-9353-9853c42337ae-host\") pod \"node-ca-rb4ph\" (UID: \"2e8325d4-12ab-4d44-9353-9853c42337ae\") " pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.639667 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639653    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-registration-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.639889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639655    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e8325d4-12ab-4d44-9353-9853c42337ae-serviceca\") pod \"node-ca-rb4ph\" (UID: \"2e8325d4-12ab-4d44-9353-9853c42337ae\") " pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.639889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639680    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/2e8325d4-12ab-4d44-9353-9853c42337ae-host\") pod \"node-ca-rb4ph\" (UID: \"2e8325d4-12ab-4d44-9353-9853c42337ae\") " pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.639889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639708    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-run-netns\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.639889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639724    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c839ad3e-92e9-4bcd-8a49-495ad550682a-tmp\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.639889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639739    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c909fa0f-361f-4896-9cf5-9b20f2e8ec94-agent-certs\") pod \"konnectivity-agent-6bf5d\" (UID: \"c909fa0f-361f-4896-9cf5-9b20f2e8ec94\") " pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:02:48.639889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639754    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jk8gm\" (UniqueName: \"kubernetes.io/projected/40eaaa5e-fe21-44f7-a769-2778ab800d3f-kube-api-access-jk8gm\") pod \"node-resolver-xlknw\" (UID: \"40eaaa5e-fe21-44f7-a769-2778ab800d3f\") " pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.639889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639790    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-system-cni-dir\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.639889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639806    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-run-systemd\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.639889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639890    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-etc-openvswitch\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639905    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639918    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-system-cni-dir\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639965    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-cnibin\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.639991    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-cnibin\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640029    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640033    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-run-netns\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640042    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-run-systemd\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640081    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zws85\" (UniqueName: \"kubernetes.io/projected/2e8325d4-12ab-4d44-9353-9853c42337ae-kube-api-access-zws85\") pod \"node-ca-rb4ph\" (UID: \"2e8325d4-12ab-4d44-9353-9853c42337ae\") " pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640102    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-system-cni-dir\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640106    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-sys-fs\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640145    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40eaaa5e-fe21-44f7-a769-2778ab800d3f-hosts-file\") pod \"node-resolver-xlknw\" (UID: \"40eaaa5e-fe21-44f7-a769-2778ab800d3f\") " pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640152    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-etc-openvswitch\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640158    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-etc-kubernetes\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640174    2589 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640190    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640189    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-etc-kubernetes\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640258    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-run-ovn-kubernetes\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.640332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640315    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-system-cni-dir\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640382    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/40eaaa5e-fe21-44f7-a769-2778ab800d3f-hosts-file\") pod \"node-resolver-xlknw\" (UID: \"40eaaa5e-fe21-44f7-a769-2778ab800d3f\") " pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640435    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-sys-fs\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640496    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-cnibin\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.640550    2589 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640200    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/2e8325d4-12ab-4d44-9353-9853c42337ae-serviceca\") pod \"node-ca-rb4ph\" (UID: \"2e8325d4-12ab-4d44-9353-9853c42337ae\") " pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.640620    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret podName:48bde41b-d00e-4bfb-8fec-0b4edbbca4f1 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:49.140600006 +0000 UTC m=+4.820285875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret") pod "global-pull-secret-syncer-89zbw" (UID: "48bde41b-d00e-4bfb-8fec-0b4edbbca4f1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640553    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-cnibin\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640666    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640782    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-ovnkube-script-lib\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640821    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640851    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-socket-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640873    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640897    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-multus-socket-dir-parent\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640908    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/973e536d-c06f-4512-b968-10b4cc35589c-iptables-alerter-script\") pod \"iptables-alerter-nk555\" (UID: \"973e536d-c06f-4512-b968-10b4cc35589c\") " pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640931    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-var-lib-cni-bin\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640969    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-var-lib-kubelet\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.643739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.640987    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dk4mv\" (UniqueName: \"kubernetes.io/projected/77c15d1f-6313-427c-b664-cf17e671cb29-kube-api-access-dk4mv\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641022    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-dbus\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641040    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-socket-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641062    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-systemd-units\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641080    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-slash\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641085    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-var-lib-cni-bin\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641101    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c909fa0f-361f-4896-9cf5-9b20f2e8ec94-konnectivity-ca\") pod \"konnectivity-agent-6bf5d\" (UID: \"c909fa0f-361f-4896-9cf5-9b20f2e8ec94\") " pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641131    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40eaaa5e-fe21-44f7-a769-2778ab800d3f-tmp-dir\") pod \"node-resolver-xlknw\" (UID: \"40eaaa5e-fe21-44f7-a769-2778ab800d3f\") " pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641154    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-var-lib-cni-multus\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641175    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-cni-binary-copy\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641195    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-sysctl-d\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641215    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-host\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641241    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-multus-cni-dir\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641264    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-run-multus-certs\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641293    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-var-lib-kubelet\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641315    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641339    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-multus-conf-dir\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641368    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-cni-netd\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.644530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641395    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-ovnkube-script-lib\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641414    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-ovnkube-config\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641454    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-systemd-units\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641465    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-var-lib-kubelet\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641516    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-host\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641557    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641607    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-sysconfig\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641641    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77c15d1f-6313-427c-b664-cf17e671cb29-multus-daemon-config\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641666    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641689    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-kubelet\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641720    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-var-lib-openvswitch\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641744    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-tuned\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641766    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-log-socket\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641774    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c909fa0f-361f-4896-9cf5-9b20f2e8ec94-konnectivity-ca\") pod \"konnectivity-agent-6bf5d\" (UID: \"c909fa0f-361f-4896-9cf5-9b20f2e8ec94\") " pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641788    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-cni-bin\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641810    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641833    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccgp6\" (UniqueName: \"kubernetes.io/projected/3d310680-2035-4659-839c-2a1357a32aba-kube-api-access-ccgp6\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:48.645252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641843    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-sysctl-d\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641857    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-hostroot\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641870    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-run-multus-certs\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641609    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-multus-socket-dir-parent\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641901    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/40eaaa5e-fe21-44f7-a769-2778ab800d3f-tmp-dir\") pod \"node-resolver-xlknw\" (UID: \"40eaaa5e-fe21-44f7-a769-2778ab800d3f\") " pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641948    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-kubelet\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641931    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-log-socket\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641978    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-var-lib-openvswitch\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642022    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-var-lib-kubelet\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642075    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641887    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5dlw\" (UniqueName: \"kubernetes.io/projected/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-kube-api-access-k5dlw\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642124    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkxf9\" (UniqueName: \"kubernetes.io/projected/973e536d-c06f-4512-b968-10b4cc35589c-kube-api-access-rkxf9\") pod \"iptables-alerter-nk555\" (UID: \"973e536d-c06f-4512-b968-10b4cc35589c\") " pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642151    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-run-netns\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642153    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-cni-binary-copy\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642156    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-multus-conf-dir\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642180    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641832    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-slash\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.646061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642244    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjmlf\" (UniqueName: \"kubernetes.io/projected/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-kube-api-access-mjmlf\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642261    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-cni-bin\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642302    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-systemd\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642327    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-hostroot\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642327    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-run\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.642368    2589 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642373    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-sys\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642389    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-host-cni-netd\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642412    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fjgx\" (UniqueName: \"kubernetes.io/projected/c839ad3e-92e9-4bcd-8a49-495ad550682a-kube-api-access-5fjgx\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642420    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-run-netns\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.642435    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs podName:3d310680-2035-4659-839c-2a1357a32aba nodeName:}" failed. No retries permitted until 2026-09-17 18:02:49.14241604 +0000 UTC m=+4.822101877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs") pod "network-metrics-daemon-jmfh6" (UID: "3d310680-2035-4659-839c-2a1357a32aba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642493    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-multus-cni-dir\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642492    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642540    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-run-k8s-cni-cncf-io\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642560    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-kubelet-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642571    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-node-log\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642605    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-lib-modules\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642613    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-run\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.646826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641883    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-var-lib-cni-multus\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642635    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-device-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642664    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/973e536d-c06f-4512-b968-10b4cc35589c-host-slash\") pod \"iptables-alerter-nk555\" (UID: \"973e536d-c06f-4512-b968-10b4cc35589c\") " pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642672    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-sysconfig\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642693    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-run-openvswitch\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642720    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-ovn-node-metrics-cert\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642724    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-sys\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642123    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-etc-selinux\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642794    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/77c15d1f-6313-427c-b664-cf17e671cb29-host-run-k8s-cni-cncf-io\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642851    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-node-log\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642867    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-lib-modules\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642896    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-systemd\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642751    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77c15d1f-6313-427c-b664-cf17e671cb29-cni-binary-copy\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642929    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-device-dir\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642943    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-kubelet-config\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642965    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/973e536d-c06f-4512-b968-10b4cc35589c-host-slash\") pod \"iptables-alerter-nk555\" (UID: \"973e536d-c06f-4512-b968-10b4cc35589c\") " pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.642971    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-env-overrides\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.647671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643021    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-modprobe-d\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643045    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-kubernetes\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643067    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-sysctl-conf\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643185    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-sysctl-conf\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643230    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-run-openvswitch\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643441    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-ovnkube-config\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643458    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-env-overrides\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.641424    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-dbus\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643531    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-modprobe-d\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643537    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77c15d1f-6313-427c-b664-cf17e671cb29-cni-binary-copy\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643533    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-kubelet-config\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643577    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-kubernetes\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.643904    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.644104    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c839ad3e-92e9-4bcd-8a49-495ad550682a-tmp\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.644112    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/77c15d1f-6313-427c-b664-cf17e671cb29-multus-daemon-config\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.644391    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c839ad3e-92e9-4bcd-8a49-495ad550682a-etc-tuned\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.644417    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c909fa0f-361f-4896-9cf5-9b20f2e8ec94-agent-certs\") pod \"konnectivity-agent-6bf5d\" (UID: \"c909fa0f-361f-4896-9cf5-9b20f2e8ec94\") " pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:02:48.648195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.646061    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-ovn-node-metrics-cert\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.651584 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.651274    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:02:48.651584 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.651294    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:02:48.651584 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.651306    2589 projected.go:196] Error preparing data for projected volume kube-api-access-rv47m for pod openshift-network-diagnostics/network-check-target-jccnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:48.651584 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:48.651364    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m podName:a6bb2b4d-1cd0-400d-b8d7-278078e402f7 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:49.151353603 +0000 UTC m=+4.831039435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rv47m" (UniqueName: "kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m") pod "network-check-target-jccnv" (UID: "a6bb2b4d-1cd0-400d-b8d7-278078e402f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:48.653240 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.653221    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jk8gm\" (UniqueName: \"kubernetes.io/projected/40eaaa5e-fe21-44f7-a769-2778ab800d3f-kube-api-access-jk8gm\") pod \"node-resolver-xlknw\" (UID: \"40eaaa5e-fe21-44f7-a769-2778ab800d3f\") " pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.653912 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.653400    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dk4mv\" (UniqueName: \"kubernetes.io/projected/77c15d1f-6313-427c-b664-cf17e671cb29-kube-api-access-dk4mv\") pod \"multus-psqh9\" (UID: \"77c15d1f-6313-427c-b664-cf17e671cb29\") " pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.653912 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.653822    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zws85\" (UniqueName: \"kubernetes.io/projected/2e8325d4-12ab-4d44-9353-9853c42337ae-kube-api-access-zws85\") pod \"node-ca-rb4ph\" (UID: \"2e8325d4-12ab-4d44-9353-9853c42337ae\") " pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.655355 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.655303    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs4cg\" (UniqueName: \"kubernetes.io/projected/0762ce87-5371-4b3e-bea0-bfdf5077a3ec-kube-api-access-xs4cg\") pod \"aws-ebs-csi-driver-node-j792q\" (UID: \"0762ce87-5371-4b3e-bea0-bfdf5077a3ec\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.658131 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.657166    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjmlf\" (UniqueName: \"kubernetes.io/projected/3ccf6be4-8960-4d93-b2eb-fce0f02e5930-kube-api-access-mjmlf\") pod \"ovnkube-node-m4trh\" (UID: \"3ccf6be4-8960-4d93-b2eb-fce0f02e5930\") " pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.658131 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.657544    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkxf9\" (UniqueName: \"kubernetes.io/projected/973e536d-c06f-4512-b968-10b4cc35589c-kube-api-access-rkxf9\") pod \"iptables-alerter-nk555\" (UID: \"973e536d-c06f-4512-b968-10b4cc35589c\") " pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.658701 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.658676    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5dlw\" (UniqueName: \"kubernetes.io/projected/7dcfddca-c2c6-4aba-aaf1-171dafa8f891-kube-api-access-k5dlw\") pod \"multus-additional-cni-plugins-t92dt\" (UID: \"7dcfddca-c2c6-4aba-aaf1-171dafa8f891\") " pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.659477 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.659080    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fjgx\" (UniqueName: \"kubernetes.io/projected/c839ad3e-92e9-4bcd-8a49-495ad550682a-kube-api-access-5fjgx\") pod \"tuned-f2nkl\" (UID: \"c839ad3e-92e9-4bcd-8a49-495ad550682a\") " pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.660440 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.660417    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccgp6\" (UniqueName: \"kubernetes.io/projected/3d310680-2035-4659-839c-2a1357a32aba-kube-api-access-ccgp6\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:48.682205 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.682156    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal" event={"ID":"3ff6991f08a902cdf413530d4bdd103e","Type":"ContainerStarted","Data":"1cba8ceb0df0ee684ebc31dea4f15c36159d0219616ba44484bf2e666e376a77"}
Sep 17 18:02:48.797828 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.797802    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q"
Sep 17 18:02:48.804077 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.804053    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rb4ph"
Sep 17 18:02:48.805449 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:48.805430    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0762ce87_5371_4b3e_bea0_bfdf5077a3ec.slice/crio-e3f41bbfec3dfea9b7231c92b59f3c294f01d29ecc98e6a1968b06dd54f21edf WatchSource:0}: Error finding container e3f41bbfec3dfea9b7231c92b59f3c294f01d29ecc98e6a1968b06dd54f21edf: Status 404 returned error can't find the container with id e3f41bbfec3dfea9b7231c92b59f3c294f01d29ecc98e6a1968b06dd54f21edf
Sep 17 18:02:48.811623 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.811607    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-xlknw"
Sep 17 18:02:48.812243 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:48.812219    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e8325d4_12ab_4d44_9353_9853c42337ae.slice/crio-bf7a12616c93b960691b21ce72b349b583acd0a5d7412e44109e07e59ba6dd07 WatchSource:0}: Error finding container bf7a12616c93b960691b21ce72b349b583acd0a5d7412e44109e07e59ba6dd07: Status 404 returned error can't find the container with id bf7a12616c93b960691b21ce72b349b583acd0a5d7412e44109e07e59ba6dd07
Sep 17 18:02:48.816411 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.816391    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:02:48.821268 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:48.821250    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40eaaa5e_fe21_44f7_a769_2778ab800d3f.slice/crio-1b5a9098c1abebcb1708e40cac209cacf1d8878feaab735bd1f3425788c68a30 WatchSource:0}: Error finding container 1b5a9098c1abebcb1708e40cac209cacf1d8878feaab735bd1f3425788c68a30: Status 404 returned error can't find the container with id 1b5a9098c1abebcb1708e40cac209cacf1d8878feaab735bd1f3425788c68a30
Sep 17 18:02:48.822218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.822201    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2nkl"
Sep 17 18:02:48.824211 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:48.824178    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ccf6be4_8960_4d93_b2eb_fce0f02e5930.slice/crio-259ff4aba59a29578e947241b8c6c14d07e2bfcce10bf0d4ec5ae9ea1d3a36d4 WatchSource:0}: Error finding container 259ff4aba59a29578e947241b8c6c14d07e2bfcce10bf0d4ec5ae9ea1d3a36d4: Status 404 returned error can't find the container with id 259ff4aba59a29578e947241b8c6c14d07e2bfcce10bf0d4ec5ae9ea1d3a36d4
Sep 17 18:02:48.825740 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.825722    2589 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 17 18:02:48.826867 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.826850    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psqh9"
Sep 17 18:02:48.830447 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:48.830425    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc839ad3e_92e9_4bcd_8a49_495ad550682a.slice/crio-b1258f8bb8175c010c1bdeb1169acdbc2ab37880425eca07ae48fdb70b8ac6b2 WatchSource:0}: Error finding container b1258f8bb8175c010c1bdeb1169acdbc2ab37880425eca07ae48fdb70b8ac6b2: Status 404 returned error can't find the container with id b1258f8bb8175c010c1bdeb1169acdbc2ab37880425eca07ae48fdb70b8ac6b2
Sep 17 18:02:48.832159 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.832073    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t92dt"
Sep 17 18:02:48.838219 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.838199    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:02:48.841127 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:48.841101    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77c15d1f_6313_427c_b664_cf17e671cb29.slice/crio-ac5c305c2e251c931931e3a41e7b58ace673a50d5574c62dfebac89102740552 WatchSource:0}: Error finding container ac5c305c2e251c931931e3a41e7b58ace673a50d5574c62dfebac89102740552: Status 404 returned error can't find the container with id ac5c305c2e251c931931e3a41e7b58ace673a50d5574c62dfebac89102740552
Sep 17 18:02:48.843356 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:48.843336    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nk555"
Sep 17 18:02:48.846562 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:48.846538    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dcfddca_c2c6_4aba_aaf1_171dafa8f891.slice/crio-bf5e0947164f18515d856f2340b95a14d072a88cc84b4d35c9e5e8389c2cbf16 WatchSource:0}: Error finding container bf5e0947164f18515d856f2340b95a14d072a88cc84b4d35c9e5e8389c2cbf16: Status 404 returned error can't find the container with id bf5e0947164f18515d856f2340b95a14d072a88cc84b4d35c9e5e8389c2cbf16
Sep 17 18:02:48.852142 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:48.851937    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc909fa0f_361f_4896_9cf5_9b20f2e8ec94.slice/crio-48c1b8291bbe33e161be4eafc3141188676b1f626dac6f4cc97d0c2c5bd3ee8e WatchSource:0}: Error finding container 48c1b8291bbe33e161be4eafc3141188676b1f626dac6f4cc97d0c2c5bd3ee8e: Status 404 returned error can't find the container with id 48c1b8291bbe33e161be4eafc3141188676b1f626dac6f4cc97d0c2c5bd3ee8e
Sep 17 18:02:48.855339 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:48.855309    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod973e536d_c06f_4512_b968_10b4cc35589c.slice/crio-33d86b701b293f1d3c193133823ebca74cf8e6d65a3c7e97737b2d1f83c40923 WatchSource:0}: Error finding container 33d86b701b293f1d3c193133823ebca74cf8e6d65a3c7e97737b2d1f83c40923: Status 404 returned error can't find the container with id 33d86b701b293f1d3c193133823ebca74cf8e6d65a3c7e97737b2d1f83c40923
Sep 17 18:02:49.146795 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.146720    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:49.146795 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.146772    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:49.146995 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.146893    2589 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:49.146995 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.146969    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs podName:3d310680-2035-4659-839c-2a1357a32aba nodeName:}" failed. No retries permitted until 2026-09-17 18:02:50.146952969 +0000 UTC m=+5.826638804 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs") pod "network-metrics-daemon-jmfh6" (UID: "3d310680-2035-4659-839c-2a1357a32aba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:49.146995 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.146902    2589 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:49.147176 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.147044    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret podName:48bde41b-d00e-4bfb-8fec-0b4edbbca4f1 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:50.147030903 +0000 UTC m=+5.826716738 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret") pod "global-pull-secret-syncer-89zbw" (UID: "48bde41b-d00e-4bfb-8fec-0b4edbbca4f1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:49.247314 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.247278    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:49.247467 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.247435    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:02:49.247467 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.247464    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:02:49.247583 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.247478    2589 projected.go:196] Error preparing data for projected volume kube-api-access-rv47m for pod openshift-network-diagnostics/network-check-target-jccnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:49.247583 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.247553    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m podName:a6bb2b4d-1cd0-400d-b8d7-278078e402f7 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:50.247532258 +0000 UTC m=+5.927218103 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rv47m" (UniqueName: "kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m") pod "network-check-target-jccnv" (UID: "a6bb2b4d-1cd0-400d-b8d7-278078e402f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:49.574155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.574105    2589 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:57:48 +0000 UTC" deadline="2028-03-25 06:23:11.675815098 +0000 UTC"
Sep 17 18:02:49.574155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.574148    2589 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13308h20m22.101671857s"
Sep 17 18:02:49.677245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.677209    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:49.677416 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.677344    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:02:49.677797 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.677776    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:49.677904 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:49.677884    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:02:49.689323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.689243    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nk555" event={"ID":"973e536d-c06f-4512-b968-10b4cc35589c","Type":"ContainerStarted","Data":"33d86b701b293f1d3c193133823ebca74cf8e6d65a3c7e97737b2d1f83c40923"}
Sep 17 18:02:49.693939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.693887    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bf5d" event={"ID":"c909fa0f-361f-4896-9cf5-9b20f2e8ec94","Type":"ContainerStarted","Data":"48c1b8291bbe33e161be4eafc3141188676b1f626dac6f4cc97d0c2c5bd3ee8e"}
Sep 17 18:02:49.695351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.695321    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psqh9" event={"ID":"77c15d1f-6313-427c-b664-cf17e671cb29","Type":"ContainerStarted","Data":"ac5c305c2e251c931931e3a41e7b58ace673a50d5574c62dfebac89102740552"}
Sep 17 18:02:49.697085 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.697040    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerStarted","Data":"259ff4aba59a29578e947241b8c6c14d07e2bfcce10bf0d4ec5ae9ea1d3a36d4"}
Sep 17 18:02:49.698525 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.698499    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rb4ph" event={"ID":"2e8325d4-12ab-4d44-9353-9853c42337ae","Type":"ContainerStarted","Data":"bf7a12616c93b960691b21ce72b349b583acd0a5d7412e44109e07e59ba6dd07"}
Sep 17 18:02:49.700349 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.700323    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t92dt" event={"ID":"7dcfddca-c2c6-4aba-aaf1-171dafa8f891","Type":"ContainerStarted","Data":"bf5e0947164f18515d856f2340b95a14d072a88cc84b4d35c9e5e8389c2cbf16"}
Sep 17 18:02:49.703921 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.703895    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2nkl" event={"ID":"c839ad3e-92e9-4bcd-8a49-495ad550682a","Type":"ContainerStarted","Data":"b1258f8bb8175c010c1bdeb1169acdbc2ab37880425eca07ae48fdb70b8ac6b2"}
Sep 17 18:02:49.707653 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.707629    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xlknw" event={"ID":"40eaaa5e-fe21-44f7-a769-2778ab800d3f","Type":"ContainerStarted","Data":"1b5a9098c1abebcb1708e40cac209cacf1d8878feaab735bd1f3425788c68a30"}
Sep 17 18:02:49.711481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:49.711457    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q" event={"ID":"0762ce87-5371-4b3e-bea0-bfdf5077a3ec","Type":"ContainerStarted","Data":"e3f41bbfec3dfea9b7231c92b59f3c294f01d29ecc98e6a1968b06dd54f21edf"}
Sep 17 18:02:49.925309 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:02:49.925263    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f4cf6b5e1eea622d2a62af388775eb4.slice/crio-ae4e5138cfa9583d7087b696da135208b5fb044106c70ec59e92461a9b049e3b WatchSource:0}: Error finding container ae4e5138cfa9583d7087b696da135208b5fb044106c70ec59e92461a9b049e3b: Status 404 returned error can't find the container with id ae4e5138cfa9583d7087b696da135208b5fb044106c70ec59e92461a9b049e3b
Sep 17 18:02:50.152358 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:50.152318    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:50.152526 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:50.152398    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:50.152589 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:50.152524    2589 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:50.152589 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:50.152582    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs podName:3d310680-2035-4659-839c-2a1357a32aba nodeName:}" failed. No retries permitted until 2026-09-17 18:02:52.152564285 +0000 UTC m=+7.832250132 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs") pod "network-metrics-daemon-jmfh6" (UID: "3d310680-2035-4659-839c-2a1357a32aba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:50.152970 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:50.152943    2589 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:50.153114 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:50.152998    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret podName:48bde41b-d00e-4bfb-8fec-0b4edbbca4f1 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:52.152983281 +0000 UTC m=+7.832669120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret") pod "global-pull-secret-syncer-89zbw" (UID: "48bde41b-d00e-4bfb-8fec-0b4edbbca4f1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:50.252941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:50.252910    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:50.253129 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:50.253084    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:02:50.253129 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:50.253110    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:02:50.253129 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:50.253123    2589 projected.go:196] Error preparing data for projected volume kube-api-access-rv47m for pod openshift-network-diagnostics/network-check-target-jccnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:50.253288 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:50.253178    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m podName:a6bb2b4d-1cd0-400d-b8d7-278078e402f7 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:52.253161548 +0000 UTC m=+7.932847393 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rv47m" (UniqueName: "kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m") pod "network-check-target-jccnv" (UID: "a6bb2b4d-1cd0-400d-b8d7-278078e402f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:50.574435 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:50.574295    2589 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-16 17:57:48 +0000 UTC" deadline="2028-04-15 04:30:09.589831601 +0000 UTC"
Sep 17 18:02:50.574435 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:50.574354    2589 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13810h27m19.015481591s"
Sep 17 18:02:50.679656 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:50.679622    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:50.679816 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:50.679753    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:02:50.718503 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:50.718398    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" event={"ID":"5f4cf6b5e1eea622d2a62af388775eb4","Type":"ContainerStarted","Data":"ae4e5138cfa9583d7087b696da135208b5fb044106c70ec59e92461a9b049e3b"}
Sep 17 18:02:50.722485 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:50.721831    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal" event={"ID":"3ff6991f08a902cdf413530d4bdd103e","Type":"ContainerStarted","Data":"933c56b105f72a1a01e161f0e99b2c0da19ea74c3f3fe4f07a2260ae9d880c05"}
Sep 17 18:02:51.678569 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:51.677893    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:51.678569 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:51.678055    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:02:51.678569 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:51.678416    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:51.678569 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:51.678517    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:02:52.169539 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:52.169497    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:52.169734 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:52.169585    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:52.169800 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:52.169733    2589 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:52.169800 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:52.169796    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs podName:3d310680-2035-4659-839c-2a1357a32aba nodeName:}" failed. No retries permitted until 2026-09-17 18:02:56.169778029 +0000 UTC m=+11.849463883 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs") pod "network-metrics-daemon-jmfh6" (UID: "3d310680-2035-4659-839c-2a1357a32aba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:52.170232 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:52.170209    2589 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:52.170344 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:52.170269    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret podName:48bde41b-d00e-4bfb-8fec-0b4edbbca4f1 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:56.170254459 +0000 UTC m=+11.849940305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret") pod "global-pull-secret-syncer-89zbw" (UID: "48bde41b-d00e-4bfb-8fec-0b4edbbca4f1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:52.270735 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:52.270691    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:52.270929 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:52.270860    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:02:52.270929 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:52.270883    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:02:52.270929 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:52.270894    2589 projected.go:196] Error preparing data for projected volume kube-api-access-rv47m for pod openshift-network-diagnostics/network-check-target-jccnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:52.271120 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:52.270942    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m podName:a6bb2b4d-1cd0-400d-b8d7-278078e402f7 nodeName:}" failed. No retries permitted until 2026-09-17 18:02:56.270929705 +0000 UTC m=+11.950615541 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rv47m" (UniqueName: "kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m") pod "network-check-target-jccnv" (UID: "a6bb2b4d-1cd0-400d-b8d7-278078e402f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:52.677236 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:52.677190    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:52.677419 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:52.677335    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:02:53.677722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:53.677181    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:53.677722 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:53.677334    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:02:53.677722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:53.677359    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:53.677722 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:53.677474    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:02:54.678071 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:54.678036    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:54.678628 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:54.678192    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:02:55.678681 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:55.678214    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:55.678681 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:55.678220    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:55.678681 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:55.678322    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:02:55.678681 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:55.678401    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:02:56.206146 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:56.206110    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:56.206328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:56.206187    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:56.206328 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:56.206288    2589 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:56.206328 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:56.206306    2589 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:56.206512 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:56.206364    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs podName:3d310680-2035-4659-839c-2a1357a32aba nodeName:}" failed. No retries permitted until 2026-09-17 18:03:04.206345526 +0000 UTC m=+19.886031372 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs") pod "network-metrics-daemon-jmfh6" (UID: "3d310680-2035-4659-839c-2a1357a32aba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:02:56.206512 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:56.206378    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret podName:48bde41b-d00e-4bfb-8fec-0b4edbbca4f1 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:04.206372109 +0000 UTC m=+19.886057941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret") pod "global-pull-secret-syncer-89zbw" (UID: "48bde41b-d00e-4bfb-8fec-0b4edbbca4f1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:02:56.307161 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:56.307098    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:56.307358 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:56.307252    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:02:56.307358 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:56.307277    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:02:56.307358 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:56.307290    2589 projected.go:196] Error preparing data for projected volume kube-api-access-rv47m for pod openshift-network-diagnostics/network-check-target-jccnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:56.307358 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:56.307347    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m podName:a6bb2b4d-1cd0-400d-b8d7-278078e402f7 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:04.307329037 +0000 UTC m=+19.987014886 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rv47m" (UniqueName: "kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m") pod "network-check-target-jccnv" (UID: "a6bb2b4d-1cd0-400d-b8d7-278078e402f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:02:56.678840 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:56.678802    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:56.679294 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:56.678923    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:02:57.677343 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:57.677257    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:57.677514 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:57.677256    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:57.677514 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:57.677431    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:02:57.677629 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:57.677548    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:02:58.677292 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:58.677249    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:02:58.677690 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:58.677397    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:02:59.677549 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:59.677514    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:02:59.678020 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:02:59.677517    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:02:59.678020 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:59.677623    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:02:59.678020 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:02:59.677738    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:00.677378 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:00.677343    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:00.677540 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:00.677477    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:01.677238 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:01.677198    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:01.677730 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:01.677330    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:01.677730 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:01.677393    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:01.677730 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:01.677515    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:02.677218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:02.677178    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:02.677385 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:02.677320    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:03.677905 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:03.677870    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:03.678402 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:03.678032    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:03.678402 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:03.678067    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:03.678402 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:03.678198    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:04.266898 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:04.266866    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:04.267119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:04.266941    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:04.267119 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:04.267088    2589 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:03:04.267242 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:04.267148    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret podName:48bde41b-d00e-4bfb-8fec-0b4edbbca4f1 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:20.267130049 +0000 UTC m=+35.946815900 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret") pod "global-pull-secret-syncer-89zbw" (UID: "48bde41b-d00e-4bfb-8fec-0b4edbbca4f1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:03:04.267570 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:04.267542    2589 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:03:04.267691 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:04.267604    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs podName:3d310680-2035-4659-839c-2a1357a32aba nodeName:}" failed. No retries permitted until 2026-09-17 18:03:20.267586155 +0000 UTC m=+35.947271992 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs") pod "network-metrics-daemon-jmfh6" (UID: "3d310680-2035-4659-839c-2a1357a32aba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:03:04.368257 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:04.368227    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:04.368411 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:04.368351    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:03:04.368411 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:04.368373    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:03:04.368411 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:04.368386    2589 projected.go:196] Error preparing data for projected volume kube-api-access-rv47m for pod openshift-network-diagnostics/network-check-target-jccnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:03:04.368528 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:04.368437    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m podName:a6bb2b4d-1cd0-400d-b8d7-278078e402f7 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:20.368424315 +0000 UTC m=+36.048110146 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rv47m" (UniqueName: "kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m") pod "network-check-target-jccnv" (UID: "a6bb2b4d-1cd0-400d-b8d7-278078e402f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:03:04.678136 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:04.678050    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:04.678541 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:04.678186    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:05.677312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:05.677280    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:05.677312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:05.677318    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:05.677561 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:05.677386    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:05.677561 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:05.677534    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:06.678344 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:06.678314    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:06.678705 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:06.678403    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:07.677954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.677921    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:07.678119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.677933    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:07.678119 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:07.678101    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:07.678210 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:07.678167    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:07.758279 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.758209    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psqh9" event={"ID":"77c15d1f-6313-427c-b664-cf17e671cb29","Type":"ContainerStarted","Data":"b51307d9eaed77bb8e09836705e563960a7dc4fc259e713a02ae90f21004222e"}
Sep 17 18:03:07.760781 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.760762    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:03:07.761134 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.761115    2589 generic.go:356] "Generic (PLEG): container finished" podID="3ccf6be4-8960-4d93-b2eb-fce0f02e5930" containerID="0889777477a96fbca9707cb0c06f1297a66af20d9dbd4c17c81c06d7acced904" exitCode=1
Sep 17 18:03:07.761204 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.761172    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerStarted","Data":"7cd4b208d4f57ca0dbe3d6d64fe7b7f694d2042fdd5d652e3042e95ce7a293fd"}
Sep 17 18:03:07.761244 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.761235    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerStarted","Data":"199eec164d6bdfcd42a01671ff0ebf675a67a9a5fca0d3860ae18068a2ac2719"}
Sep 17 18:03:07.761296 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.761285    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerDied","Data":"0889777477a96fbca9707cb0c06f1297a66af20d9dbd4c17c81c06d7acced904"}
Sep 17 18:03:07.761331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.761325    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerStarted","Data":"eafb05e0fa07a889d4a599aa7ebf95c2e0b09a5db70391667da6edb9c0011ee6"}
Sep 17 18:03:07.763060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.763040    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2nkl" event={"ID":"c839ad3e-92e9-4bcd-8a49-495ad550682a","Type":"ContainerStarted","Data":"bfd9a2b842281eb1630103f69582c407a5de01c7e566f481287c99ca1ff46d9e"}
Sep 17 18:03:07.803447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.803398    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-142-105.ec2.internal" podStartSLOduration=19.803375754 podStartE2EDuration="19.803375754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:02:50.750074416 +0000 UTC m=+6.429760270" watchObservedRunningTime="2026-09-17 18:03:07.803375754 +0000 UTC m=+23.483061608"
Sep 17 18:03:07.854642 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.854597    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-psqh9" podStartSLOduration=2.492025393 podStartE2EDuration="20.8545805s" totalImagesPullingTime="18.362555107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:07.799590461 +0000 UTC m=+23.479276323" watchObservedRunningTime="2026-09-17 18:03:07.8545805 +0000 UTC m=+23.534266354"
Sep 17 18:03:07.854948 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:07.854928    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f2nkl" podStartSLOduration=3.520041115 podStartE2EDuration="21.85492269s" totalImagesPullingTime="18.334881575s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:07.854371056 +0000 UTC m=+23.534056900" watchObservedRunningTime="2026-09-17 18:03:07.85492269 +0000 UTC m=+23.534608557"
Sep 17 18:03:08.677744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.677720    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:08.677846 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:08.677829    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:08.766015 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.765977    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q" event={"ID":"0762ce87-5371-4b3e-bea0-bfdf5077a3ec","Type":"ContainerStarted","Data":"04adfed323db6edb021b168eab6b5a7dbafd275a9a08bb08f24ddbda6131d41a"}
Sep 17 18:03:08.767504 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.767484    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nk555" event={"ID":"973e536d-c06f-4512-b968-10b4cc35589c","Type":"ContainerStarted","Data":"cc7147ce9b7b6d240ce6da733d477b47b44b32458943db0f906e75d6e992c5de"}
Sep 17 18:03:08.768961 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.768941    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6bf5d" event={"ID":"c909fa0f-361f-4896-9cf5-9b20f2e8ec94","Type":"ContainerStarted","Data":"a5a0ee1dce169e9f5c7f0e307655b47512b4e95c7bc508170e7ae97653dbb9eb"}
Sep 17 18:03:08.771821 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.771805    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:03:08.774786 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.772222    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerStarted","Data":"bc32f9f09d5bfbfebced839fcc95635708db8a4b0709b180fe54b3ce498ef4a2"}
Sep 17 18:03:08.775198 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.775167    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerStarted","Data":"1de6f18f64e2b5080f640dfb24bcf589564ee16c1cb8ec29f34c7cd1f0d5fb4b"}
Sep 17 18:03:08.776399 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.776374    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rb4ph" event={"ID":"2e8325d4-12ab-4d44-9353-9853c42337ae","Type":"ContainerStarted","Data":"48bd9569a19aa942f3e6e873e768603937f816dce6638407abcd925aafe83ad6"}
Sep 17 18:03:08.777812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.777789    2589 generic.go:356] "Generic (PLEG): container finished" podID="5f4cf6b5e1eea622d2a62af388775eb4" containerID="dbd2abf34bb585fcd70b00028413b7740aaad71f8362cbd8a433d85bbfc64064" exitCode=0
Sep 17 18:03:08.777946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.777854    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" event={"ID":"5f4cf6b5e1eea622d2a62af388775eb4","Type":"ContainerDied","Data":"dbd2abf34bb585fcd70b00028413b7740aaad71f8362cbd8a433d85bbfc64064"}
Sep 17 18:03:08.779422 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.779400    2589 generic.go:356] "Generic (PLEG): container finished" podID="7dcfddca-c2c6-4aba-aaf1-171dafa8f891" containerID="cbd2c0bb4fc561559790ad475384ce6bb98b6df5dd823142919449fe563123d3" exitCode=0
Sep 17 18:03:08.779506 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.779479    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t92dt" event={"ID":"7dcfddca-c2c6-4aba-aaf1-171dafa8f891","Type":"ContainerDied","Data":"cbd2c0bb4fc561559790ad475384ce6bb98b6df5dd823142919449fe563123d3"}
Sep 17 18:03:08.780966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.780947    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-xlknw" event={"ID":"40eaaa5e-fe21-44f7-a769-2778ab800d3f","Type":"ContainerStarted","Data":"576c98e4eea3d7bdf82e56b7f404e0eb2def0eb590cfc2d3543c7622a48a5d94"}
Sep 17 18:03:08.802044 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.801988    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nk555" podStartSLOduration=3.492674348 podStartE2EDuration="21.801977108s" totalImagesPullingTime="18.30930276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:08.801709409 +0000 UTC m=+24.481395263" watchObservedRunningTime="2026-09-17 18:03:08.801977108 +0000 UTC m=+24.481662962"
Sep 17 18:03:08.854819 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.854776    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6bf5d" podStartSLOduration=3.598605538 podStartE2EDuration="21.854762812s" totalImagesPullingTime="18.256157274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:08.854596445 +0000 UTC m=+24.534282300" watchObservedRunningTime="2026-09-17 18:03:08.854762812 +0000 UTC m=+24.534448663"
Sep 17 18:03:08.939886 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.939834    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-xlknw" podStartSLOduration=4.653533451 podStartE2EDuration="22.939819267s" totalImagesPullingTime="18.286285816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:08.939649212 +0000 UTC m=+24.619335066" watchObservedRunningTime="2026-09-17 18:03:08.939819267 +0000 UTC m=+24.619505118"
Sep 17 18:03:08.940086 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:08.939922    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rb4ph" podStartSLOduration=4.644250437 podStartE2EDuration="22.939913355s" totalImagesPullingTime="18.295662918s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:08.917320448 +0000 UTC m=+24.597006301" watchObservedRunningTime="2026-09-17 18:03:08.939913355 +0000 UTC m=+24.619599210"
Sep 17 18:03:09.677969 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:09.677934    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:09.677969 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:09.677957    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:09.678331 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:09.678120    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:09.678331 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:09.678232    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:09.786584 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:09.786545    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" event={"ID":"5f4cf6b5e1eea622d2a62af388775eb4","Type":"ContainerStarted","Data":"4788f660c6e4cde8995312926a35f0482553f2ec20344fe705c368743b8666b2"}
Sep 17 18:03:10.137240 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:10.136932    2589 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 17 18:03:10.602832 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:10.602737    2589 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-17T18:03:10.136956365Z","UUID":"5eb4a963-c418-4e0b-b041-4b9f837455bd","Handler":null,"Name":"","Endpoint":""}
Sep 17 18:03:10.604644 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:10.604377    2589 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 17 18:03:10.604644 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:10.604415    2589 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 17 18:03:10.677555 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:10.677522    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:10.677710 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:10.677633    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:10.791537 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:10.791494    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q" event={"ID":"0762ce87-5371-4b3e-bea0-bfdf5077a3ec","Type":"ContainerStarted","Data":"35d1221adb60c185037d28bc0c03c9cc78f3825e8bb0b889d66b0d2f82548aba"}
Sep 17 18:03:10.795210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:10.795190    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:03:10.795581 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:10.795562    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerStarted","Data":"b0298a82bf7d44a2bd611c11592ef1b4a6bd7b0c62fb29325bc05bf1e328b463"}
Sep 17 18:03:11.677607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:11.677573    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:11.677791 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:11.677680    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:11.677791 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:11.677705    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:11.677791 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:11.677752    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:11.802284 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:11.802245    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q" event={"ID":"0762ce87-5371-4b3e-bea0-bfdf5077a3ec","Type":"ContainerStarted","Data":"fd43efc6a10f71b9a37000c446753b1a25d0c13eb4abcbaeb0483e970261f9f2"}
Sep 17 18:03:11.823982 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:11.823927    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-j792q" podStartSLOduration=5.01795467 podStartE2EDuration="25.823908634s" totalImagesPullingTime="20.805953964s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:46 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:11.823703062 +0000 UTC m=+27.503389106" watchObservedRunningTime="2026-09-17 18:03:11.823908634 +0000 UTC m=+27.503594489"
Sep 17 18:03:11.824219 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:11.824182    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-142-105.ec2.internal" podStartSLOduration=23.824172711 podStartE2EDuration="23.824172711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:09.81317068 +0000 UTC m=+25.492856534" watchObservedRunningTime="2026-09-17 18:03:11.824172711 +0000 UTC m=+27.503858566"
Sep 17 18:03:12.343470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:12.343433    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:03:12.344073 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:12.344055    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:03:12.677218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:12.677183    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:12.677382 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:12.677354    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:12.808884 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:12.808856    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:03:12.809294 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:12.809267    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerStarted","Data":"328970bf179906659eace5e8b748dac21ebaf450b614232a9f7cf3c335af3a2b"}
Sep 17 18:03:12.809856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:12.809690    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:03:12.809856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:12.809791    2589 scope.go:117] "RemoveContainer" containerID="0889777477a96fbca9707cb0c06f1297a66af20d9dbd4c17c81c06d7acced904"
Sep 17 18:03:12.827090 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:12.827069    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:03:13.677589 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:13.677548    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:13.677783 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:13.677605    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:13.677783 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:13.677699    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:13.677896 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:13.677827    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:14.314802 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.314779    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-89zbw"]
Sep 17 18:03:14.315461 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.314882    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:14.315461 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:14.314959    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:14.321626 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.321591    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jmfh6"]
Sep 17 18:03:14.321735 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.321680    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:14.321789 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:14.321770    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:14.334889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.334864    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jccnv"]
Sep 17 18:03:14.334965 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.334941    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:14.335047 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:14.335031    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:14.817491 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.817464    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:03:14.817815 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.817790    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" event={"ID":"3ccf6be4-8960-4d93-b2eb-fce0f02e5930","Type":"ContainerStarted","Data":"7ddb74e38f31146a35b9384f04b851429f34694efcdb8e1482785f14daa8a5ca"}
Sep 17 18:03:14.817958 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.817945    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:03:14.818036 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.818026    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:03:14.819668 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.819645    2589 generic.go:356] "Generic (PLEG): container finished" podID="7dcfddca-c2c6-4aba-aaf1-171dafa8f891" containerID="3ffd2a5832bb35c263942dcc6859b43e3f1a674d017df07a8d506b167c2b61ca" exitCode=0
Sep 17 18:03:14.819763 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.819694    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t92dt" event={"ID":"7dcfddca-c2c6-4aba-aaf1-171dafa8f891","Type":"ContainerDied","Data":"3ffd2a5832bb35c263942dcc6859b43e3f1a674d017df07a8d506b167c2b61ca"}
Sep 17 18:03:14.833614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.833589    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:03:14.862241 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:14.862202    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh" podStartSLOduration=10.500528002 podStartE2EDuration="28.862188665s" totalImagesPullingTime="18.361660663s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:14.86095904 +0000 UTC m=+30.540644895" watchObservedRunningTime="2026-09-17 18:03:14.862188665 +0000 UTC m=+30.541874524"
Sep 17 18:03:15.677766 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:15.677727    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:15.678293 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:15.677775    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:15.678293 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:15.677865    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:15.678293 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:15.677995    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:16.008168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:16.008130    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:03:16.008345 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:16.008279    2589 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Sep 17 18:03:16.008807 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:16.008780    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6bf5d"
Sep 17 18:03:16.678976 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:16.678776    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:16.679382 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:16.678995    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:16.827090 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:16.827051    2589 generic.go:356] "Generic (PLEG): container finished" podID="7dcfddca-c2c6-4aba-aaf1-171dafa8f891" containerID="1369005c7b19b33b16add07f7fa94f9ab0be1a14f61cad1e668f51e08e90c0a7" exitCode=0
Sep 17 18:03:16.827250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:16.827131    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t92dt" event={"ID":"7dcfddca-c2c6-4aba-aaf1-171dafa8f891","Type":"ContainerDied","Data":"1369005c7b19b33b16add07f7fa94f9ab0be1a14f61cad1e668f51e08e90c0a7"}
Sep 17 18:03:17.678252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:17.678209    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:17.678252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:17.678248    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:17.678508 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:17.678334    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:17.678508 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:17.678444    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:18.677515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:18.677483    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:18.677876 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:18.677607    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-89zbw" podUID="48bde41b-d00e-4bfb-8fec-0b4edbbca4f1"
Sep 17 18:03:18.834242 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:18.834205    2589 generic.go:356] "Generic (PLEG): container finished" podID="7dcfddca-c2c6-4aba-aaf1-171dafa8f891" containerID="481db12b6ce1351ee367ff669e86f04d9f52a491afb2b92c6018b5b99df12470" exitCode=0
Sep 17 18:03:18.834428 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:18.834262    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t92dt" event={"ID":"7dcfddca-c2c6-4aba-aaf1-171dafa8f891","Type":"ContainerDied","Data":"481db12b6ce1351ee367ff669e86f04d9f52a491afb2b92c6018b5b99df12470"}
Sep 17 18:03:19.677657 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:19.677620    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:19.678101 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:19.677620    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:19.678101 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:19.677729    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jccnv" podUID="a6bb2b4d-1cd0-400d-b8d7-278078e402f7"
Sep 17 18:03:19.678101 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:19.677790    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-jmfh6" podUID="3d310680-2035-4659-839c-2a1357a32aba"
Sep 17 18:03:20.204026 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.203982    2589 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-142-105.ec2.internal" event="NodeReady"
Sep 17 18:03:20.204160 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.204148    2589 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 17 18:03:20.254264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.254230    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"]
Sep 17 18:03:20.274910 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.274879    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.274910 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.274908    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-clrtz"]
Sep 17 18:03:20.280126 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.280105    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kgd2r\""
Sep 17 18:03:20.280974 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.280951    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 17 18:03:20.281094 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.280989    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 17 18:03:20.281094 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.281054    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 17 18:03:20.285251 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.285232    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 17 18:03:20.289869 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.289851    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:20.289962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.289895    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:20.290016 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.289980    2589 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:03:20.290016 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.289986    2589 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 17 18:03:20.290078 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.290041    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs podName:3d310680-2035-4659-839c-2a1357a32aba nodeName:}" failed. No retries permitted until 2026-09-17 18:03:52.290029774 +0000 UTC m=+67.969715618 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs") pod "network-metrics-daemon-jmfh6" (UID: "3d310680-2035-4659-839c-2a1357a32aba") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 17 18:03:20.290078 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.290053    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret podName:48bde41b-d00e-4bfb-8fec-0b4edbbca4f1 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:52.290047325 +0000 UTC m=+67.969733157 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret") pod "global-pull-secret-syncer-89zbw" (UID: "48bde41b-d00e-4bfb-8fec-0b4edbbca4f1") : object "kube-system"/"original-pull-secret" not registered
Sep 17 18:03:20.299778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.299759    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"]
Sep 17 18:03:20.299862 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.299787    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mmxsl"]
Sep 17 18:03:20.299899 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.299869    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.302629 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.302613    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 17 18:03:20.302711 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.302614    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dfk9\""
Sep 17 18:03:20.302747 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.302714    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 17 18:03:20.322758 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.322741    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mmxsl"]
Sep 17 18:03:20.322835 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.322767    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-clrtz"]
Sep 17 18:03:20.322835 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.322834    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mmxsl"
Sep 17 18:03:20.325597 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.325542    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-fht8w\""
Sep 17 18:03:20.325597 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.325556    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 17 18:03:20.325597 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.325563    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 17 18:03:20.325780 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.325605    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 17 18:03:20.390379 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390339    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:20.390543 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390400    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-image-registry-private-configuration\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.390543 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390429    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxh79\" (UniqueName: \"kubernetes.io/projected/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-kube-api-access-sxh79\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.390543 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390455    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdfv8\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-kube-api-access-kdfv8\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.390543 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390473    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-bound-sa-token\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.390543 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390497    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4jt9\" (UniqueName: \"kubernetes.io/projected/b1c403c8-5032-4a13-9413-cc9bc512fd3f-kube-api-access-m4jt9\") pod \"ingress-canary-mmxsl\" (UID: \"b1c403c8-5032-4a13-9413-cc9bc512fd3f\") " pod="openshift-ingress-canary/ingress-canary-mmxsl"
Sep 17 18:03:20.390543 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.390511    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 17 18:03:20.390543 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.390535    2589 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 17 18:03:20.390543 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.390548    2589 projected.go:196] Error preparing data for projected volume kube-api-access-rv47m for pod openshift-network-diagnostics/network-check-target-jccnv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:03:20.390946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390556    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-installation-pull-secrets\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.390946 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.390629    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m podName:a6bb2b4d-1cd0-400d-b8d7-278078e402f7 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:52.390609964 +0000 UTC m=+68.070295813 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rv47m" (UniqueName: "kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m") pod "network-check-target-jccnv" (UID: "a6bb2b4d-1cd0-400d-b8d7-278078e402f7") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 17 18:03:20.390946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390717    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-ca-trust-extracted\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.390946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390743    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-trusted-ca\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.390946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390789    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.390946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390839    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-config-volume\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.390946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390861    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.390946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390881    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-tmp-dir\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.390946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390918    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-certificates\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.391417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.390954    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c403c8-5032-4a13-9413-cc9bc512fd3f-cert\") pod \"ingress-canary-mmxsl\" (UID: \"b1c403c8-5032-4a13-9413-cc9bc512fd3f\") " pod="openshift-ingress-canary/ingress-canary-mmxsl"
Sep 17 18:03:20.491774 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.491738    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-ca-trust-extracted\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.491774 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.491777    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-trusted-ca\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.492092 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492052    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.492236 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492115    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-config-volume\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.492236 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492145    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.492236 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.492170    2589 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:03:20.492236 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492209    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-ca-trust-extracted\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.492236 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.492237    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls podName:fc1f8942-acd9-43aa-a7e4-e5cccc1d0359 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:20.992217511 +0000 UTC m=+36.671903346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls") pod "dns-default-clrtz" (UID: "fc1f8942-acd9-43aa-a7e4-e5cccc1d0359") : secret "dns-default-metrics-tls" not found
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492174    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-tmp-dir\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.492252    2589 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.492266    2589 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd97fdbb8-6xzsf: secret "image-registry-tls" not found
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492276    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-certificates\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492306    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c403c8-5032-4a13-9413-cc9bc512fd3f-cert\") pod \"ingress-canary-mmxsl\" (UID: \"b1c403c8-5032-4a13-9413-cc9bc512fd3f\") " pod="openshift-ingress-canary/ingress-canary-mmxsl"
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.492328    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls podName:cd7c076c-56fd-4f5e-890e-13a07d2f6d51 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:20.992312532 +0000 UTC m=+36.671998387 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls") pod "image-registry-7dd97fdbb8-6xzsf" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51") : secret "image-registry-tls" not found
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492390    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-image-registry-private-configuration\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492424    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxh79\" (UniqueName: \"kubernetes.io/projected/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-kube-api-access-sxh79\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492458    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdfv8\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-kube-api-access-kdfv8\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.492501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492488    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-bound-sa-token\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.492960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492512    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4jt9\" (UniqueName: \"kubernetes.io/projected/b1c403c8-5032-4a13-9413-cc9bc512fd3f-kube-api-access-m4jt9\") pod \"ingress-canary-mmxsl\" (UID: \"b1c403c8-5032-4a13-9413-cc9bc512fd3f\") " pod="openshift-ingress-canary/ingress-canary-mmxsl"
Sep 17 18:03:20.492960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492544    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-installation-pull-secrets\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.492960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492647    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-config-volume\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.492960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492830    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-trusted-ca\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.492960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.492904    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-tmp-dir\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.496800 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.496774    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-installation-pull-secrets\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.496800 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.496786    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-image-registry-private-configuration\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.496937 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.496838    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b1c403c8-5032-4a13-9413-cc9bc512fd3f-cert\") pod \"ingress-canary-mmxsl\" (UID: \"b1c403c8-5032-4a13-9413-cc9bc512fd3f\") " pod="openshift-ingress-canary/ingress-canary-mmxsl"
Sep 17 18:03:20.501452 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.501404    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-certificates\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.501708 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.501690    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxh79\" (UniqueName: \"kubernetes.io/projected/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-kube-api-access-sxh79\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.504101 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.504081    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdfv8\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-kube-api-access-kdfv8\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.504210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.504195    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4jt9\" (UniqueName: \"kubernetes.io/projected/b1c403c8-5032-4a13-9413-cc9bc512fd3f-kube-api-access-m4jt9\") pod \"ingress-canary-mmxsl\" (UID: \"b1c403c8-5032-4a13-9413-cc9bc512fd3f\") " pod="openshift-ingress-canary/ingress-canary-mmxsl"
Sep 17 18:03:20.505208 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.505158    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-bound-sa-token\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.632880 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.632844    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mmxsl"
Sep 17 18:03:20.678265 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.678234    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:20.681331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.681305    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:03:20.789062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.788818    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:20.972488 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.972459    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mmxsl"]
Sep 17 18:03:20.977750 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:03:20.977718    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c403c8_5032_4a13_9413_cc9bc512fd3f.slice/crio-034603e998e59d8076b37e49b96532ce734eff63d6b2d5afd8fa9161895af3bc WatchSource:0}: Error finding container 034603e998e59d8076b37e49b96532ce734eff63d6b2d5afd8fa9161895af3bc: Status 404 returned error can't find the container with id 034603e998e59d8076b37e49b96532ce734eff63d6b2d5afd8fa9161895af3bc
Sep 17 18:03:20.988951 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.988923    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:20.997119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.997093    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:20.997222 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:20.997130    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:20.997268 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.997231    2589 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:03:20.997268 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.997235    2589 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:03:20.997268 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.997249    2589 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd97fdbb8-6xzsf: secret "image-registry-tls" not found
Sep 17 18:03:20.997528 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.997290    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls podName:fc1f8942-acd9-43aa-a7e4-e5cccc1d0359 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:21.997271511 +0000 UTC m=+37.676957359 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls") pod "dns-default-clrtz" (UID: "fc1f8942-acd9-43aa-a7e4-e5cccc1d0359") : secret "dns-default-metrics-tls" not found
Sep 17 18:03:20.997528 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:20.997307    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls podName:cd7c076c-56fd-4f5e-890e-13a07d2f6d51 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:21.997298601 +0000 UTC m=+37.676984433 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls") pod "image-registry-7dd97fdbb8-6xzsf" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51") : secret "image-registry-tls" not found
Sep 17 18:03:21.677942 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:21.677900    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:21.678314 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:21.678301    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:21.681132 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:21.681108    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:03:21.682521 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:21.682501    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2v7h2\""
Sep 17 18:03:21.682780 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:21.682756    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qc24z\""
Sep 17 18:03:21.682968 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:21.682940    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:03:21.683186 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:21.683151    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:03:21.844476 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:21.844398    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mmxsl" event={"ID":"b1c403c8-5032-4a13-9413-cc9bc512fd3f","Type":"ContainerStarted","Data":"034603e998e59d8076b37e49b96532ce734eff63d6b2d5afd8fa9161895af3bc"}
Sep 17 18:03:22.004201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:22.004167    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:22.004393 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:22.004227    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:22.004393 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:22.004350    2589 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:03:22.004393 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:22.004378    2589 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:03:22.004393 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:22.004392    2589 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd97fdbb8-6xzsf: secret "image-registry-tls" not found
Sep 17 18:03:22.004538 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:22.004427    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls podName:fc1f8942-acd9-43aa-a7e4-e5cccc1d0359 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:24.004406759 +0000 UTC m=+39.684092595 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls") pod "dns-default-clrtz" (UID: "fc1f8942-acd9-43aa-a7e4-e5cccc1d0359") : secret "dns-default-metrics-tls" not found
Sep 17 18:03:22.004538 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:22.004446    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls podName:cd7c076c-56fd-4f5e-890e-13a07d2f6d51 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:24.00443717 +0000 UTC m=+39.684123008 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls") pod "image-registry-7dd97fdbb8-6xzsf" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51") : secret "image-registry-tls" not found
Sep 17 18:03:24.018974 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:24.018943    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:24.019703 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:24.018994    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:24.019703 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:24.019130    2589 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:03:24.019703 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:24.019198    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls podName:fc1f8942-acd9-43aa-a7e4-e5cccc1d0359 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:28.019180385 +0000 UTC m=+43.698866234 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls") pod "dns-default-clrtz" (UID: "fc1f8942-acd9-43aa-a7e4-e5cccc1d0359") : secret "dns-default-metrics-tls" not found
Sep 17 18:03:24.019703 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:24.019136    2589 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:03:24.019703 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:24.019228    2589 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd97fdbb8-6xzsf: secret "image-registry-tls" not found
Sep 17 18:03:24.019703 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:24.019279    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls podName:cd7c076c-56fd-4f5e-890e-13a07d2f6d51 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:28.019267949 +0000 UTC m=+43.698953780 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls") pod "image-registry-7dd97fdbb8-6xzsf" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51") : secret "image-registry-tls" not found
Sep 17 18:03:24.676618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:24.676584    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:24.853145 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:24.853108    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mmxsl" event={"ID":"b1c403c8-5032-4a13-9413-cc9bc512fd3f","Type":"ContainerStarted","Data":"507dd3e6a9439545884a8bbb219de116dca94d8565bf29bbf24315e25b19f592"}
Sep 17 18:03:24.858142 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:24.856591    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t92dt" event={"ID":"7dcfddca-c2c6-4aba-aaf1-171dafa8f891","Type":"ContainerStarted","Data":"a674ef1fd322c48508e8c730ebac86f927c33a2a42b8ff10d3b8ff768c9ce76f"}
Sep 17 18:03:24.873178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:24.873126    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mmxsl" podStartSLOduration=1.120972331 podStartE2EDuration="4.873110787s" totalImagesPullingTime="3.752138456s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:24.871840719 +0000 UTC m=+40.551526573" watchObservedRunningTime="2026-09-17 18:03:24.873110787 +0000 UTC m=+40.552796639"
Sep 17 18:03:25.861480 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:25.861449    2589 generic.go:356] "Generic (PLEG): container finished" podID="7dcfddca-c2c6-4aba-aaf1-171dafa8f891" containerID="a674ef1fd322c48508e8c730ebac86f927c33a2a42b8ff10d3b8ff768c9ce76f" exitCode=0
Sep 17 18:03:25.861842 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:25.861523    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t92dt" event={"ID":"7dcfddca-c2c6-4aba-aaf1-171dafa8f891","Type":"ContainerDied","Data":"a674ef1fd322c48508e8c730ebac86f927c33a2a42b8ff10d3b8ff768c9ce76f"}
Sep 17 18:03:25.898523 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:25.898494    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:26.866732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:26.866699    2589 generic.go:356] "Generic (PLEG): container finished" podID="7dcfddca-c2c6-4aba-aaf1-171dafa8f891" containerID="d0f941f37b6914b012bcc5364b1e9f10f6f1eaf946e7f1a2dcb0684a52004f44" exitCode=0
Sep 17 18:03:26.867223 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:26.866761    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t92dt" event={"ID":"7dcfddca-c2c6-4aba-aaf1-171dafa8f891","Type":"ContainerDied","Data":"d0f941f37b6914b012bcc5364b1e9f10f6f1eaf946e7f1a2dcb0684a52004f44"}
Sep 17 18:03:27.575986 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:27.575954    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:27.652425 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:27.652394    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:27.751664 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:27.751629    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:27.872185 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:27.872111    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t92dt" event={"ID":"7dcfddca-c2c6-4aba-aaf1-171dafa8f891","Type":"ContainerStarted","Data":"e231bddf6f64b66f957aae4d85e7d18ad5902005c3a9b46bd23aabc5afdcc261"}
Sep 17 18:03:27.914163 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:27.914117    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t92dt" podStartSLOduration=8.744621502 podStartE2EDuration="40.914101807s" totalImagesPullingTime="32.169480305s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:27.912221259 +0000 UTC m=+43.591907113" watchObservedRunningTime="2026-09-17 18:03:27.914101807 +0000 UTC m=+43.593787660"
Sep 17 18:03:28.053299 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:28.053267    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:28.053299 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:28.053305    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:28.053486 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:28.053411    2589 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:03:28.053486 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:28.053416    2589 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:03:28.053486 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:28.053433    2589 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd97fdbb8-6xzsf: secret "image-registry-tls" not found
Sep 17 18:03:28.053486 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:28.053473    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls podName:fc1f8942-acd9-43aa-a7e4-e5cccc1d0359 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:36.053456706 +0000 UTC m=+51.733142538 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls") pod "dns-default-clrtz" (UID: "fc1f8942-acd9-43aa-a7e4-e5cccc1d0359") : secret "dns-default-metrics-tls" not found
Sep 17 18:03:28.053486 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:28.053487    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls podName:cd7c076c-56fd-4f5e-890e-13a07d2f6d51 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:36.053481299 +0000 UTC m=+51.733167131 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls") pod "image-registry-7dd97fdbb8-6xzsf" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51") : secret "image-registry-tls" not found
Sep 17 18:03:28.058321 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:28.058298    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:28.161416 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:28.161339    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:28.655866 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:28.655831    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:28.725055 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:28.724994    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:29.049850 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:29.049821    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:29.143320 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:29.143279    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:29.220247 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:29.220212    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:29.579932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:29.579902    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:35.861527 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:35.861496    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:35.937097 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:35.937062    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:36.115879 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:36.115792    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:36.115879 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:36.115840    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:36.116128 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:36.115953    2589 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:03:36.116128 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:36.116045    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls podName:fc1f8942-acd9-43aa-a7e4-e5cccc1d0359 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:52.116028431 +0000 UTC m=+67.795714277 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls") pod "dns-default-clrtz" (UID: "fc1f8942-acd9-43aa-a7e4-e5cccc1d0359") : secret "dns-default-metrics-tls" not found
Sep 17 18:03:36.116128 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:36.115954    2589 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:03:36.116128 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:36.116068    2589 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd97fdbb8-6xzsf: secret "image-registry-tls" not found
Sep 17 18:03:36.116279 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:36.116137    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls podName:cd7c076c-56fd-4f5e-890e-13a07d2f6d51 nodeName:}" failed. No retries permitted until 2026-09-17 18:03:52.116120561 +0000 UTC m=+67.795806409 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls") pod "image-registry-7dd97fdbb8-6xzsf" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51") : secret "image-registry-tls" not found
Sep 17 18:03:36.932282 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:36.932250    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:37.026556 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:37.026527    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:46.837705 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:46.837673    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m4trh"
Sep 17 18:03:47.022284 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:47.022247    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:47.111257 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:47.111168    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:47.722496 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:47.722451    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:47.850579 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:47.850542    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:48.018544 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:48.018509    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:48.116926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:48.116893    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:52.130102 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.130068    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:03:52.130102 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.130106    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:03:52.130583 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:52.130204    2589 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 17 18:03:52.130583 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:52.130214    2589 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7dd97fdbb8-6xzsf: secret "image-registry-tls" not found
Sep 17 18:03:52.130583 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:52.130222    2589 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 17 18:03:52.130583 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:52.130277    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls podName:cd7c076c-56fd-4f5e-890e-13a07d2f6d51 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:24.130261924 +0000 UTC m=+99.809947769 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls") pod "image-registry-7dd97fdbb8-6xzsf" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51") : secret "image-registry-tls" not found
Sep 17 18:03:52.130583 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:52.130288    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls podName:fc1f8942-acd9-43aa-a7e4-e5cccc1d0359 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:24.130283182 +0000 UTC m=+99.809969014 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls") pod "dns-default-clrtz" (UID: "fc1f8942-acd9-43aa-a7e4-e5cccc1d0359") : secret "dns-default-metrics-tls" not found
Sep 17 18:03:52.331350 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.331295    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:52.331540 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.331377    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:03:52.334140 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.334120    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 17 18:03:52.334220 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.334200    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 17 18:03:52.342201 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:52.342179    2589 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 17 18:03:52.342250 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:03:52.342245    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs podName:3d310680-2035-4659-839c-2a1357a32aba nodeName:}" failed. No retries permitted until 2026-09-17 18:04:56.342227507 +0000 UTC m=+132.021913363 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs") pod "network-metrics-daemon-jmfh6" (UID: "3d310680-2035-4659-839c-2a1357a32aba") : secret "metrics-daemon-secret" not found
Sep 17 18:03:52.345105 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.345087    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/48bde41b-d00e-4bfb-8fec-0b4edbbca4f1-original-pull-secret\") pod \"global-pull-secret-syncer-89zbw\" (UID: \"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1\") " pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:52.432324 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.432230    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:52.434991 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.434970    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 17 18:03:52.445970 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.445947    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 17 18:03:52.456665 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.456646    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rv47m\" (UniqueName: \"kubernetes.io/projected/a6bb2b4d-1cd0-400d-b8d7-278078e402f7-kube-api-access-rv47m\") pod \"network-check-target-jccnv\" (UID: \"a6bb2b4d-1cd0-400d-b8d7-278078e402f7\") " pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:52.489566 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.489537    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-89zbw"
Sep 17 18:03:52.593962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.593935    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2v7h2\""
Sep 17 18:03:52.601709 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.601689    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:52.775290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.775259    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-89zbw"]
Sep 17 18:03:52.778514 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:03:52.778479    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bde41b_d00e_4bfb_8fec_0b4edbbca4f1.slice/crio-5c5ac33a3805f5d95590f525f19e195064cd2b8d079b5c06bbef28b95590b4d4 WatchSource:0}: Error finding container 5c5ac33a3805f5d95590f525f19e195064cd2b8d079b5c06bbef28b95590b4d4: Status 404 returned error can't find the container with id 5c5ac33a3805f5d95590f525f19e195064cd2b8d079b5c06bbef28b95590b4d4
Sep 17 18:03:52.892048 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.891996    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jccnv"]
Sep 17 18:03:52.895555 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:03:52.895535    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6bb2b4d_1cd0_400d_b8d7_278078e402f7.slice/crio-0c425f4574126c6e6f90a467c748ac091b8af6a0d334a23362969e4fab74b3cc WatchSource:0}: Error finding container 0c425f4574126c6e6f90a467c748ac091b8af6a0d334a23362969e4fab74b3cc: Status 404 returned error can't find the container with id 0c425f4574126c6e6f90a467c748ac091b8af6a0d334a23362969e4fab74b3cc
Sep 17 18:03:52.930468 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.930436    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jccnv" event={"ID":"a6bb2b4d-1cd0-400d-b8d7-278078e402f7","Type":"ContainerStarted","Data":"0c425f4574126c6e6f90a467c748ac091b8af6a0d334a23362969e4fab74b3cc"}
Sep 17 18:03:52.931614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:52.931594    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-89zbw" event={"ID":"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1","Type":"ContainerStarted","Data":"5c5ac33a3805f5d95590f525f19e195064cd2b8d079b5c06bbef28b95590b4d4"}
Sep 17 18:03:54.912226 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:54.912191    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:54.994495 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:54.994423    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:57.016166 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:57.015989    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:57.133079 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:57.133039    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:03:57.948352 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:57.948324    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jccnv" event={"ID":"a6bb2b4d-1cd0-400d-b8d7-278078e402f7","Type":"ContainerStarted","Data":"87edb6e67ea6fec1dcaac419ac074877395e7ab875fb1c4f6d1ec164fcc3e757"}
Sep 17 18:03:57.948552 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:57.948529    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:03:57.968203 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:57.968165    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jccnv" podStartSLOduration=66.994961664 podStartE2EDuration="1m11.968150761s" totalImagesPullingTime="4.973189097s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:57.96706789 +0000 UTC m=+73.646753745" watchObservedRunningTime="2026-09-17 18:03:57.968150761 +0000 UTC m=+73.647836606"
Sep 17 18:03:58.951774 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:03:58.951730    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-89zbw" event={"ID":"48bde41b-d00e-4bfb-8fec-0b4edbbca4f1","Type":"ContainerStarted","Data":"5ffde59b32624c6bab175bdcb2d01fc2f6e8ae6d5a6a51c9674e6b7f85c59ed9"}
Sep 17 18:04:00.011173 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.011136    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.112166 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.112131    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.548407 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.548372    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.558559 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.558510    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-89zbw" podStartSLOduration=68.463968888 podStartE2EDuration="1m13.558491513s" totalImagesPullingTime="5.094522625s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:03:58.974261988 +0000 UTC m=+74.653947841" watchObservedRunningTime="2026-09-17 18:04:00.558491513 +0000 UTC m=+76.238177380"
Sep 17 18:04:00.558703 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.558689    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-5xq4s"]
Sep 17 18:04:00.560756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.560740    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.565733 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.565709    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 17 18:04:00.566068 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.566053    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 17 18:04:00.567054 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.567038    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 17 18:04:00.567128 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.567072    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 17 18:04:00.570957 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.570896    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-87jlz\""
Sep 17 18:04:00.590595 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.590573    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxhfm\" (UniqueName: \"kubernetes.io/projected/9f032f75-5f73-4847-889d-98d6e9b06785-kube-api-access-sxhfm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.590595 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.590602    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.590735 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.590673    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.590735 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.590726    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f032f75-5f73-4847-889d-98d6e9b06785-crio-socket\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.590803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.590744    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f032f75-5f73-4847-889d-98d6e9b06785-data-volume\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.597528 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.597501    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xq4s"]
Sep 17 18:04:00.691904 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.691874    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxhfm\" (UniqueName: \"kubernetes.io/projected/9f032f75-5f73-4847-889d-98d6e9b06785-kube-api-access-sxhfm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.691904 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.691905    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.692150 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:00.691984    2589 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:04:00.692150 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:00.692059    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.192044283 +0000 UTC m=+76.871730115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : configmap "kube-rbac-proxy" not found
Sep 17 18:04:00.692150 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.692134    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.692267 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.692204    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f032f75-5f73-4847-889d-98d6e9b06785-crio-socket\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.692267 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.692231    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f032f75-5f73-4847-889d-98d6e9b06785-data-volume\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.692343 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.692284    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/9f032f75-5f73-4847-889d-98d6e9b06785-crio-socket\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.692343 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:00.692289    2589 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:00.692416 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:00.692396    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:01.192382169 +0000 UTC m=+76.872068024 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:00.692526 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.692511    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f032f75-5f73-4847-889d-98d6e9b06785-data-volume\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:00.695948 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.695912    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:00.741498 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:00.741469    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxhfm\" (UniqueName: \"kubernetes.io/projected/9f032f75-5f73-4847-889d-98d6e9b06785-kube-api-access-sxhfm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:01.195776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:01.195730    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:01.196184 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:01.195795    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:01.196184 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:01.195876    2589 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:04:01.196184 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:01.195887    2589 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:01.196184 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:01.195945    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:02.195931008 +0000 UTC m=+77.875616840 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:01.196184 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:01.195958    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:02.195951715 +0000 UTC m=+77.875637547 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : configmap "kube-rbac-proxy" not found
Sep 17 18:04:02.202466 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:02.202423    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:02.202901 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:02.202485    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:02.202901 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:02.202541    2589 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:04:02.202901 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:02.202590    2589 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:02.202901 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:02.202611    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.202594579 +0000 UTC m=+79.882280429 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : configmap "kube-rbac-proxy" not found
Sep 17 18:04:02.202901 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:02.202651    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:04.202634055 +0000 UTC m=+79.882319889 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:02.631702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:02.631673    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xlknw_40eaaa5e-fe21-44f7-a769-2778ab800d3f/dns-node-resolver/0.log"
Sep 17 18:04:02.734372 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:02.734332    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:02.838593 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:02.838561    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:03.430925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:03.430900    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rb4ph_2e8325d4-12ab-4d44-9353-9853c42337ae/node-ca/0.log"
Sep 17 18:04:04.217248 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:04.217199    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:04.217436 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:04.217268    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:04.217436 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:04.217331    2589 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:04:04.217436 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:04.217369    2589 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:04.217436 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:04.217403    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:08.217387664 +0000 UTC m=+83.897073513 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : configmap "kube-rbac-proxy" not found
Sep 17 18:04:04.217436 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:04.217420    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:08.217412607 +0000 UTC m=+83.897098439 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:04.236070 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:04.236045    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mmxsl_b1c403c8-5032-4a13-9413-cc9bc512fd3f/serve-healthcheck-canary/0.log"
Sep 17 18:04:06.603434 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.603405    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"]
Sep 17 18:04:06.608867 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.608844    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.611735 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.611693    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:04:06.611882 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.611798    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 17 18:04:06.611882 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.611808    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 17 18:04:06.612112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.612098    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 17 18:04:06.612788 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.612772    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-dm7rp\""
Sep 17 18:04:06.632802 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.632778    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"]
Sep 17 18:04:06.636262 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.636240    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a7691fb-f931-4285-bce6-68555253947f-serving-cert\") pod \"service-ca-operator-57cdc4b768-6rgf2\" (UID: \"7a7691fb-f931-4285-bce6-68555253947f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.636372 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.636310    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2dfw\" (UniqueName: \"kubernetes.io/projected/7a7691fb-f931-4285-bce6-68555253947f-kube-api-access-d2dfw\") pod \"service-ca-operator-57cdc4b768-6rgf2\" (UID: \"7a7691fb-f931-4285-bce6-68555253947f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.636372 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.636333    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a7691fb-f931-4285-bce6-68555253947f-config\") pod \"service-ca-operator-57cdc4b768-6rgf2\" (UID: \"7a7691fb-f931-4285-bce6-68555253947f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.736677 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.736651    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a7691fb-f931-4285-bce6-68555253947f-serving-cert\") pod \"service-ca-operator-57cdc4b768-6rgf2\" (UID: \"7a7691fb-f931-4285-bce6-68555253947f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.736849 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.736706    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2dfw\" (UniqueName: \"kubernetes.io/projected/7a7691fb-f931-4285-bce6-68555253947f-kube-api-access-d2dfw\") pod \"service-ca-operator-57cdc4b768-6rgf2\" (UID: \"7a7691fb-f931-4285-bce6-68555253947f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.736849 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.736730    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a7691fb-f931-4285-bce6-68555253947f-config\") pod \"service-ca-operator-57cdc4b768-6rgf2\" (UID: \"7a7691fb-f931-4285-bce6-68555253947f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.737351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.737333    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7a7691fb-f931-4285-bce6-68555253947f-config\") pod \"service-ca-operator-57cdc4b768-6rgf2\" (UID: \"7a7691fb-f931-4285-bce6-68555253947f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.739031 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.738991    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7a7691fb-f931-4285-bce6-68555253947f-serving-cert\") pod \"service-ca-operator-57cdc4b768-6rgf2\" (UID: \"7a7691fb-f931-4285-bce6-68555253947f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.745278 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.745257    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2dfw\" (UniqueName: \"kubernetes.io/projected/7a7691fb-f931-4285-bce6-68555253947f-kube-api-access-d2dfw\") pod \"service-ca-operator-57cdc4b768-6rgf2\" (UID: \"7a7691fb-f931-4285-bce6-68555253947f\") " pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:06.918104 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:06.918032    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"
Sep 17 18:04:07.186911 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:07.186837    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2"]
Sep 17 18:04:07.190190 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:07.190161    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7691fb_f931_4285_bce6_68555253947f.slice/crio-f2130ffb0ffdb0196cf706a4f5207d57df7b4a74f50d80dc098f445e3384d881 WatchSource:0}: Error finding container f2130ffb0ffdb0196cf706a4f5207d57df7b4a74f50d80dc098f445e3384d881: Status 404 returned error can't find the container with id f2130ffb0ffdb0196cf706a4f5207d57df7b4a74f50d80dc098f445e3384d881
Sep 17 18:04:07.979741 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:07.979704    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2" event={"ID":"7a7691fb-f931-4285-bce6-68555253947f","Type":"ContainerStarted","Data":"f2130ffb0ffdb0196cf706a4f5207d57df7b4a74f50d80dc098f445e3384d881"}
Sep 17 18:04:08.248034 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:08.247934    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:08.248034 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:08.247998    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:08.248270 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:08.248114    2589 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 17 18:04:08.248270 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:08.248120    2589 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:08.248270 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:08.248197    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.248175414 +0000 UTC m=+91.927861261 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:08.248270 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:08.248220    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:16.248208647 +0000 UTC m=+91.927894509 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : configmap "kube-rbac-proxy" not found
Sep 17 18:04:09.986042 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:09.985981    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2" event={"ID":"7a7691fb-f931-4285-bce6-68555253947f","Type":"ContainerStarted","Data":"5263099a00aa526251130537273762caf58c9350c8cb208ac86ec8bcea5f76de"}
Sep 17 18:04:10.007140 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:10.007094    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2" podStartSLOduration=1.848047526 podStartE2EDuration="4.00707926s" totalImagesPullingTime="2.159031734s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:10.005581481 +0000 UTC m=+85.685267335" watchObservedRunningTime="2026-09-17 18:04:10.00707926 +0000 UTC m=+85.686765114"
Sep 17 18:04:10.716114 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:10.716077    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:10.809055 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:10.809024    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:11.011257 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.011222    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:11.105321 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.105293    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z"]
Sep 17 18:04:11.107616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.107599    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z"
Sep 17 18:04:11.112840 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.112818    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-l8wgt\""
Sep 17 18:04:11.113890 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.113763    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 17 18:04:11.118331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.118311    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 17 18:04:11.126568 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.126544    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z"]
Sep 17 18:04:11.163235 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.163209    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:11.171196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.171176    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7xqh\" (UniqueName: \"kubernetes.io/projected/dd39f199-8168-4a5e-b58f-4b3d7f11aace-kube-api-access-b7xqh\") pod \"migrator-76794cbd6c-xwv7z\" (UID: \"dd39f199-8168-4a5e-b58f-4b3d7f11aace\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z"
Sep 17 18:04:11.252430 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.252401    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:11.272373 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.272318    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7xqh\" (UniqueName: \"kubernetes.io/projected/dd39f199-8168-4a5e-b58f-4b3d7f11aace-kube-api-access-b7xqh\") pod \"migrator-76794cbd6c-xwv7z\" (UID: \"dd39f199-8168-4a5e-b58f-4b3d7f11aace\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z"
Sep 17 18:04:11.281465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.281436    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7xqh\" (UniqueName: \"kubernetes.io/projected/dd39f199-8168-4a5e-b58f-4b3d7f11aace-kube-api-access-b7xqh\") pod \"migrator-76794cbd6c-xwv7z\" (UID: \"dd39f199-8168-4a5e-b58f-4b3d7f11aace\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z"
Sep 17 18:04:11.416580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.416547    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z"
Sep 17 18:04:11.427459 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.427431    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"]
Sep 17 18:04:11.430109 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.430083    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:11.432901 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.432880    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-5dlsd\""
Sep 17 18:04:11.433039 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.432976    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 17 18:04:11.434220 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.434200    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 17 18:04:11.442186 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.442161    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"]
Sep 17 18:04:11.474494 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.474464    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f49e94b-b01e-4085-940f-f2048dd17044-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:11.474601 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.474515    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:11.550370 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.550282    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:11.575317 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.575290    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f49e94b-b01e-4085-940f-f2048dd17044-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:11.575453 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.575324    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:11.575453 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:11.575432    2589 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:04:11.575564 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:11.575489    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert podName:9f49e94b-b01e-4085-940f-f2048dd17044 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:12.075472639 +0000 UTC m=+87.755158471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fpbjm" (UID: "9f49e94b-b01e-4085-940f-f2048dd17044") : secret "networking-console-plugin-cert" not found
Sep 17 18:04:11.575975 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.575951    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f49e94b-b01e-4085-940f-f2048dd17044-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:11.652204 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.652173    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:11.689550 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.689528    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z"]
Sep 17 18:04:11.692233 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:11.692209    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd39f199_8168_4a5e_b58f_4b3d7f11aace.slice/crio-1f0d55bcba7a16f10bc06ebd2e3dd517f24644506089c712b8f75eed27a47c6f WatchSource:0}: Error finding container 1f0d55bcba7a16f10bc06ebd2e3dd517f24644506089c712b8f75eed27a47c6f: Status 404 returned error can't find the container with id 1f0d55bcba7a16f10bc06ebd2e3dd517f24644506089c712b8f75eed27a47c6f
Sep 17 18:04:11.983489 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.983381    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:11.992200 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:11.992169    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z" event={"ID":"dd39f199-8168-4a5e-b58f-4b3d7f11aace","Type":"ContainerStarted","Data":"1f0d55bcba7a16f10bc06ebd2e3dd517f24644506089c712b8f75eed27a47c6f"}
Sep 17 18:04:12.079230 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:12.079185    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:12.079578 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:12.079343    2589 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:04:12.079578 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:12.079401    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert podName:9f49e94b-b01e-4085-940f-f2048dd17044 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:13.079388647 +0000 UTC m=+88.759074483 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fpbjm" (UID: "9f49e94b-b01e-4085-940f-f2048dd17044") : secret "networking-console-plugin-cert" not found
Sep 17 18:04:12.084397 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:12.084377    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:12.275491 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:12.275453    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:12.372666 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:12.372632    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:12.997154 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:12.997069    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z" event={"ID":"dd39f199-8168-4a5e-b58f-4b3d7f11aace","Type":"ContainerStarted","Data":"588a17d78a350de2030e7c17abf7c37109f7664b3c4ebad9d924f886f32b863f"}
Sep 17 18:04:12.997154 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:12.997122    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z" event={"ID":"dd39f199-8168-4a5e-b58f-4b3d7f11aace","Type":"ContainerStarted","Data":"9d63eb44fd88c2a11837afe2f95a458acd45e8aad73403b4a9210332b0b4403f"}
Sep 17 18:04:13.019665 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:13.019623    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-xwv7z" podStartSLOduration=0.977790724 podStartE2EDuration="2.019608266s" totalImagesPullingTime="1.041817542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:13.017542762 +0000 UTC m=+88.697228616" watchObservedRunningTime="2026-09-17 18:04:13.019608266 +0000 UTC m=+88.699294157"
Sep 17 18:04:13.087972 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:13.087951    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:13.088283 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:13.088104    2589 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:04:13.088283 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:13.088171    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert podName:9f49e94b-b01e-4085-940f-f2048dd17044 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:15.088156159 +0000 UTC m=+90.767842016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fpbjm" (UID: "9f49e94b-b01e-4085-940f-f2048dd17044") : secret "networking-console-plugin-cert" not found
Sep 17 18:04:13.168353 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:13.168321    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:13.298643 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:13.298606    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:13.670805 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:13.670721    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:13.904404 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:13.904369    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:14.005750 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:14.005726    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:14.147810 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:14.147776    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:14.230091 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:14.230059    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:15.104064 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:15.104029    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:15.104239 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:15.104151    2589 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:04:15.104239 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:15.104208    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert podName:9f49e94b-b01e-4085-940f-f2048dd17044 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:19.104193987 +0000 UTC m=+94.783879818 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fpbjm" (UID: "9f49e94b-b01e-4085-940f-f2048dd17044") : secret "networking-console-plugin-cert" not found
Sep 17 18:04:15.200668 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:15.200630    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:16.312179 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:16.312150    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:16.312544 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:16.312222    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:16.312544 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:16.312309    2589 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:16.312544 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:16.312380    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls podName:9f032f75-5f73-4847-889d-98d6e9b06785 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:32.312363714 +0000 UTC m=+107.992049550 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls") pod "insights-runtime-extractor-5xq4s" (UID: "9f032f75-5f73-4847-889d-98d6e9b06785") : secret "insights-runtime-extractor-tls" not found
Sep 17 18:04:16.312801 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:16.312783    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/9f032f75-5f73-4847-889d-98d6e9b06785-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:17.463189 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:17.463156    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:17.562481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:17.562446    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:17.789037 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:17.788992    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:17.903072 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:17.903040    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:18.003607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:18.003564    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:18.159529 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:18.159443    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:18.757692 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:18.757655    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:19.135347 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:19.135256    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:19.135509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:19.135415    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:19.135509 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:19.135448    2589 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 17 18:04:19.135604 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:04:19.135519    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert podName:9f49e94b-b01e-4085-940f-f2048dd17044 nodeName:}" failed. No retries permitted until 2026-09-17 18:04:27.135500844 +0000 UTC m=+102.815186677 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert") pod "networking-console-plugin-6446c8d75d-fpbjm" (UID: "9f49e94b-b01e-4085-940f-f2048dd17044") : secret "networking-console-plugin-cert" not found
Sep 17 18:04:19.811334 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:19.811291    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:19.943258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:19.943225    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:20.209378 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:20.209296    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:20.538460 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:20.538430    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:20.746678 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:20.746645    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:21.144340 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:21.144307    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:21.298535 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:21.298495    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:21.408508 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:21.408430    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:21.745855 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:21.745774    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:22.135968 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:22.135934    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:22.214661 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:22.214628    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:22.346238 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:22.346199    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:22.792346 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:22.792309    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:23.242671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:23.242588    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:23.343818 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:23.343786    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:23.736370 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:23.736300    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:24.135673 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.135634    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:24.175664 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.175626    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:04:24.175756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.175692    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:04:24.178366 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.178323    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"image-registry-7dd97fdbb8-6xzsf\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") " pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:04:24.178461 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.178380    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fc1f8942-acd9-43aa-a7e4-e5cccc1d0359-metrics-tls\") pod \"dns-default-clrtz\" (UID: \"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359\") " pod="openshift-dns/dns-default-clrtz"
Sep 17 18:04:24.188702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.188685    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-kgd2r\""
Sep 17 18:04:24.196944 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.196926    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:04:24.210677 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.210659    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7dfk9\""
Sep 17 18:04:24.218882 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.218863    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-clrtz"
Sep 17 18:04:24.337197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.337163    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:24.472462 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.472435    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"]
Sep 17 18:04:24.475126 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:24.475086    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7c076c_56fd_4f5e_890e_13a07d2f6d51.slice/crio-6b0caf653957c6de5f1aceee8290d0f06e7281a2800433720c77c8475ebbc300 WatchSource:0}: Error finding container 6b0caf653957c6de5f1aceee8290d0f06e7281a2800433720c77c8475ebbc300: Status 404 returned error can't find the container with id 6b0caf653957c6de5f1aceee8290d0f06e7281a2800433720c77c8475ebbc300
Sep 17 18:04:24.500047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.500021    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-clrtz"]
Sep 17 18:04:24.505928 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:24.505901    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc1f8942_acd9_43aa_a7e4_e5cccc1d0359.slice/crio-e75695ecb84a7b640fa596d780af8948d00c4256b02b6bd012f5aef7987155dc WatchSource:0}: Error finding container e75695ecb84a7b640fa596d780af8948d00c4256b02b6bd012f5aef7987155dc: Status 404 returned error can't find the container with id e75695ecb84a7b640fa596d780af8948d00c4256b02b6bd012f5aef7987155dc
Sep 17 18:04:24.729264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:24.729181    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:25.034591 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:25.034551    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clrtz" event={"ID":"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359","Type":"ContainerStarted","Data":"e75695ecb84a7b640fa596d780af8948d00c4256b02b6bd012f5aef7987155dc"}
Sep 17 18:04:25.036070 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:25.036043    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf" event={"ID":"cd7c076c-56fd-4f5e-890e-13a07d2f6d51","Type":"ContainerStarted","Data":"79f67b5ae3410b146e5c823779c1eb2f91b94df528b427a2020accbf5f36bc90"}
Sep 17 18:04:25.036157 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:25.036084    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf" event={"ID":"cd7c076c-56fd-4f5e-890e-13a07d2f6d51","Type":"ContainerStarted","Data":"6b0caf653957c6de5f1aceee8290d0f06e7281a2800433720c77c8475ebbc300"}
Sep 17 18:04:25.036205 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:25.036181    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:04:25.062091 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:25.062047    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf" podStartSLOduration=98.062031241 podStartE2EDuration="1m38.062031241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:25.060435713 +0000 UTC m=+100.740121561" watchObservedRunningTime="2026-09-17 18:04:25.062031241 +0000 UTC m=+100.741717095"
Sep 17 18:04:25.340129 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:25.340036    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:25.940204 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:25.940163    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:26.144553 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:26.144519    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:26.354541 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:26.354491    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:26.743786 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:26.743693    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:27.201392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:27.201364    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:27.204279 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:27.204264    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f49e94b-b01e-4085-940f-f2048dd17044-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-fpbjm\" (UID: \"9f49e94b-b01e-4085-940f-f2048dd17044\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:27.338944 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:27.338699    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"
Sep 17 18:04:27.345436 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:27.345399    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:27.451952 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:27.451825    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:27.582897 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:27.582856    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:27.663741 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:27.663720    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm"]
Sep 17 18:04:27.665880 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:27.665851    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f49e94b_b01e_4085_940f_f2048dd17044.slice/crio-69ff51e13808f0965bbc757bdbcbac2e87ba9438e8fcec6152519857947692d4 WatchSource:0}: Error finding container 69ff51e13808f0965bbc757bdbcbac2e87ba9438e8fcec6152519857947692d4: Status 404 returned error can't find the container with id 69ff51e13808f0965bbc757bdbcbac2e87ba9438e8fcec6152519857947692d4
Sep 17 18:04:27.815650 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:27.815623    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:27.920949 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:27.920906    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:28.021867 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.021831    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:28.046653 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.046621    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm" event={"ID":"9f49e94b-b01e-4085-940f-f2048dd17044","Type":"ContainerStarted","Data":"69ff51e13808f0965bbc757bdbcbac2e87ba9438e8fcec6152519857947692d4"}
Sep 17 18:04:28.048425 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.048399    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clrtz" event={"ID":"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359","Type":"ContainerStarted","Data":"7b7dcdf715c736718a3a6cc8277fd0ca62984021b51b3626a76666f4cbafbc5b"}
Sep 17 18:04:28.048531 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.048450    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-clrtz" event={"ID":"fc1f8942-acd9-43aa-a7e4-e5cccc1d0359","Type":"ContainerStarted","Data":"cf7a8f515179ac9169b7a8551528c91b49d370671265d0ba8cb2930a9a155f43"}
Sep 17 18:04:28.048531 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.048517    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-clrtz"
Sep 17 18:04:28.079050 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.078963    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-clrtz" podStartSLOduration=65.409126934 podStartE2EDuration="1m8.078945577s" totalImagesPullingTime="2.669818643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:03:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:28.077311189 +0000 UTC m=+103.756997043" watchObservedRunningTime="2026-09-17 18:04:28.078945577 +0000 UTC m=+103.758631433"
Sep 17 18:04:28.123724 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.123703    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:28.210220 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.210188    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:28.300267 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.300242    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:28.384753 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.384674    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:28.954350 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:28.954315    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jccnv"
Sep 17 18:04:29.303914 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:29.303872    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:29.503775 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:29.503741    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:29.832983 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:29.832946    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:30.009256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:30.009214    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:30.057126 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:30.057087    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm" event={"ID":"9f49e94b-b01e-4085-940f-f2048dd17044","Type":"ContainerStarted","Data":"af33762fc1a5debd563647d14d716c589d9a63475e8802f881e6ebbfa4a5357d"}
Sep 17 18:04:30.082854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:30.082807    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-fpbjm" podStartSLOduration=17.420731099 podStartE2EDuration="19.082791236s" totalImagesPullingTime="1.662060137s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:30.082451568 +0000 UTC m=+105.762137424" watchObservedRunningTime="2026-09-17 18:04:30.082791236 +0000 UTC m=+105.762477090"
Sep 17 18:04:31.068748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:31.068709    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:31.172741 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:31.172701    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:31.271623 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:31.271594    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:32.151778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:32.151747    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:32.336620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:32.336588    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:32.339067 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:32.339039    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/9f032f75-5f73-4847-889d-98d6e9b06785-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-5xq4s\" (UID: \"9f032f75-5f73-4847-889d-98d6e9b06785\") " pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:32.370239 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:32.370206    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-5xq4s"
Sep 17 18:04:32.645197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:32.645168    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-5xq4s"]
Sep 17 18:04:32.648254 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:32.648226    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f032f75_5f73_4847_889d_98d6e9b06785.slice/crio-050030b54ec224dca43cf0fb70904ffec1b595e6e80e9295194b12a11eefcebf WatchSource:0}: Error finding container 050030b54ec224dca43cf0fb70904ffec1b595e6e80e9295194b12a11eefcebf: Status 404 returned error can't find the container with id 050030b54ec224dca43cf0fb70904ffec1b595e6e80e9295194b12a11eefcebf
Sep 17 18:04:33.072375 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:33.072338    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xq4s" event={"ID":"9f032f75-5f73-4847-889d-98d6e9b06785","Type":"ContainerStarted","Data":"038798d49904bc9f0c289ed3ba941cdcee12d267477c86002a31736a3c109be4"}
Sep 17 18:04:33.072546 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:33.072413    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xq4s" event={"ID":"9f032f75-5f73-4847-889d-98d6e9b06785","Type":"ContainerStarted","Data":"050030b54ec224dca43cf0fb70904ffec1b595e6e80e9295194b12a11eefcebf"}
Sep 17 18:04:34.077117 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:34.077079    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xq4s" event={"ID":"9f032f75-5f73-4847-889d-98d6e9b06785","Type":"ContainerStarted","Data":"dcfc719f9ef6520339ed813c0b8d2b687db40270eff61fd7e5431fb970f14a5c"}
Sep 17 18:04:35.082064 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:35.082030    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-5xq4s" event={"ID":"9f032f75-5f73-4847-889d-98d6e9b06785","Type":"ContainerStarted","Data":"6d74cf11ed4c24d8cbe4907b1d7bdff9ca88329fdb2fe7867c01a2754760e324"}
Sep 17 18:04:35.112260 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:35.112223    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-5xq4s" podStartSLOduration=32.859705903 podStartE2EDuration="35.112209571s" totalImagesPullingTime="2.252503668s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:04:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:35.111478462 +0000 UTC m=+110.791164318" watchObservedRunningTime="2026-09-17 18:04:35.112209571 +0000 UTC m=+110.791895425"
Sep 17 18:04:35.998146 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:35.998112    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"]
Sep 17 18:04:36.000218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.000201    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.003224 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.003193    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 17 18:04:36.003350 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.003259    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 17 18:04:36.004072 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.004057    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 17 18:04:36.004207 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.004105    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 17 18:04:36.004252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.004238    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 17 18:04:36.004339 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.004319    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 17 18:04:36.004380 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.004334    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 17 18:04:36.027274 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.027253    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"]
Sep 17 18:04:36.164347 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.164318    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.164730 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.164350    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.164730 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.164420    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-hub\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.164730 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.164453    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.164730 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.164568    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-ca\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.164730 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.164607    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzdt5\" (UniqueName: \"kubernetes.io/projected/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-kube-api-access-vzdt5\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.265196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.265162    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.265363 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.265206    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.265363 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.265237    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-hub\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.265363 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.265261    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.265509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.265425    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-ca\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.265509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.265465    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzdt5\" (UniqueName: \"kubernetes.io/projected/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-kube-api-access-vzdt5\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.266294 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.266256    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-additional-service-ca\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.267895 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.267869    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-ca\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.268069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.268051    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.268131 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.268054    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.268131 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.268103    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-hub\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.309447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.309420    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:36.311934 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.311915    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzdt5\" (UniqueName: \"kubernetes.io/projected/7a1cf1f4-1302-4cbd-ae8c-813b845f08cc-kube-api-access-vzdt5\") pod \"cluster-proxy-proxy-agent-58994cbff7-lxn8t\" (UID: \"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.322760 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.322741    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"
Sep 17 18:04:36.488910 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.488872    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:36.611223 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.611148    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t"]
Sep 17 18:04:36.614640 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:36.614603    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a1cf1f4_1302_4cbd_ae8c_813b845f08cc.slice/crio-9ec99f70ea3216441d9b7927e241c5c44c98f59e0006808d1f7a1ba24c36a123 WatchSource:0}: Error finding container 9ec99f70ea3216441d9b7927e241c5c44c98f59e0006808d1f7a1ba24c36a123: Status 404 returned error can't find the container with id 9ec99f70ea3216441d9b7927e241c5c44c98f59e0006808d1f7a1ba24c36a123
Sep 17 18:04:36.616113 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:36.616071    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:37.089418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:37.089376    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t" event={"ID":"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc","Type":"ContainerStarted","Data":"9ec99f70ea3216441d9b7927e241c5c44c98f59e0006808d1f7a1ba24c36a123"}
Sep 17 18:04:38.053998 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:38.053965    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-clrtz"
Sep 17 18:04:40.293856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:40.293823    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:40.398184 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:40.398128    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:40.669247 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:40.669163    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:42.108877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:42.108838    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t" event={"ID":"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc","Type":"ContainerStarted","Data":"a01078cdfb9d9ee274e20ad06bf427a30c952f9c2d71fd4fdbbd5f1dad479628"}
Sep 17 18:04:42.108877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:42.108884    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t" event={"ID":"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc","Type":"ContainerStarted","Data":"2ff4d1e15725f5ea1a2a2b88f1617733ff5019f82199512ca570b54daf54f26c"}
Sep 17 18:04:42.109312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:42.108897    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t" event={"ID":"7a1cf1f4-1302-4cbd-ae8c-813b845f08cc","Type":"ContainerStarted","Data":"089e09733fe78b16d37d13695db1b59934465d4e6a6bbd0e0c5d7d28fcec55e2"}
Sep 17 18:04:42.125897 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:42.125844    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:42.136561 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:42.135526    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-58994cbff7-lxn8t" podStartSLOduration=2.323239437 podStartE2EDuration="7.135506038s" totalImagesPullingTime="4.812266601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:42.13325558 +0000 UTC m=+117.812941435" watchObservedRunningTime="2026-09-17 18:04:42.135506038 +0000 UTC m=+117.815191892"
Sep 17 18:04:42.226990 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:42.226956    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:42.669923 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:42.669889    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:44.134188 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:44.134160    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:44.201459 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:44.201413    2589 patch_prober.go:28] interesting pod/image-registry-7dd97fdbb8-6xzsf container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 17 18:04:44.201608 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:44.201486    2589 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf" podUID="cd7c076c-56fd-4f5e-890e-13a07d2f6d51" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:04:44.210839 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:44.210781    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:44.612118 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:44.612081    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:44.858557 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:44.858523    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:45.022465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.022425    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:45.069086 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.069061    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"]
Sep 17 18:04:45.071719 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.071700    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.075067 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.075050    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 17 18:04:45.076187 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.076157    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-ggh6c\""
Sep 17 18:04:45.076291 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.076203    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 17 18:04:45.076291 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.076220    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 17 18:04:45.076291 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.076229    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 17 18:04:45.077424 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.077400    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 17 18:04:45.090375 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.090343    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"]
Sep 17 18:04:45.174372 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.174333    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hqlpd"]
Sep 17 18:04:45.176618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.176602    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.182854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.182804    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 17 18:04:45.182854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.182827    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 17 18:04:45.182854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.182831    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 17 18:04:45.183090 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.182921    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-27csq\""
Sep 17 18:04:45.203920 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.203883    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:45.232844 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.232813    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdzfd\" (UniqueName: \"kubernetes.io/projected/7cc5f99a-a7a9-4949-81f9-3572052afd7f-kube-api-access-qdzfd\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.232995 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.232865    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cc5f99a-a7a9-4949-81f9-3572052afd7f-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.232995 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.232959    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cc5f99a-a7a9-4949-81f9-3572052afd7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.233105 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.233031    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5f99a-a7a9-4949-81f9-3572052afd7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.323542 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.323449    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:45.333575 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333555    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cc5f99a-a7a9-4949-81f9-3572052afd7f-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.333653 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333585    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-tls\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.333653 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333616    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-accelerators-collector-config\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.333748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333670    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cc5f99a-a7a9-4949-81f9-3572052afd7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.333748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333706    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46e3b407-8f77-463c-a7bd-b12c146dd498-metrics-client-ca\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.333748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333735    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-wtmp\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.333902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333755    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.333902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333776    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46e3b407-8f77-463c-a7bd-b12c146dd498-sys\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.333902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333795    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5f99a-a7a9-4949-81f9-3572052afd7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.333902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333827    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46e3b407-8f77-463c-a7bd-b12c146dd498-root\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.333902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333876    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdzfd\" (UniqueName: \"kubernetes.io/projected/7cc5f99a-a7a9-4949-81f9-3572052afd7f-kube-api-access-qdzfd\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.334168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333927    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n95h4\" (UniqueName: \"kubernetes.io/projected/46e3b407-8f77-463c-a7bd-b12c146dd498-kube-api-access-n95h4\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.334168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.333975    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-textfile\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.334452 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.334348    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7cc5f99a-a7a9-4949-81f9-3572052afd7f-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.336186 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.336166    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7cc5f99a-a7a9-4949-81f9-3572052afd7f-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.336336 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.336320    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cc5f99a-a7a9-4949-81f9-3572052afd7f-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.342635 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.342616    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdzfd\" (UniqueName: \"kubernetes.io/projected/7cc5f99a-a7a9-4949-81f9-3572052afd7f-kube-api-access-qdzfd\") pod \"openshift-state-metrics-86c5bc79d6-hb4k5\" (UID: \"7cc5f99a-a7a9-4949-81f9-3572052afd7f\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.380481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.380457    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"
Sep 17 18:04:45.434425 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.434395    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-accelerators-collector-config\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.434560 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.434447    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46e3b407-8f77-463c-a7bd-b12c146dd498-metrics-client-ca\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.434560 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.434488    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-wtmp\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.434560 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.434515    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.434560 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.434542    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46e3b407-8f77-463c-a7bd-b12c146dd498-sys\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.434778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.434571    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46e3b407-8f77-463c-a7bd-b12c146dd498-root\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.434778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.434596    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n95h4\" (UniqueName: \"kubernetes.io/projected/46e3b407-8f77-463c-a7bd-b12c146dd498-kube-api-access-n95h4\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.434778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.434636    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-textfile\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.434778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.434672    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-tls\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.435959 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.435062    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-accelerators-collector-config\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.435959 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.435130    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/46e3b407-8f77-463c-a7bd-b12c146dd498-sys\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.435959 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.435219    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-wtmp\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.435959 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.435457    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46e3b407-8f77-463c-a7bd-b12c146dd498-metrics-client-ca\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.435959 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.435517    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/46e3b407-8f77-463c-a7bd-b12c146dd498-root\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.435959 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.435741    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-textfile\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.436904 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.436791    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:45.437929 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.437894    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.439716 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.439694    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/46e3b407-8f77-463c-a7bd-b12c146dd498-node-exporter-tls\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.448519 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.448500    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n95h4\" (UniqueName: \"kubernetes.io/projected/46e3b407-8f77-463c-a7bd-b12c146dd498-kube-api-access-n95h4\") pod \"node-exporter-hqlpd\" (UID: \"46e3b407-8f77-463c-a7bd-b12c146dd498\") " pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.485136 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.485108    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hqlpd"
Sep 17 18:04:45.494753 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:45.494733    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46e3b407_8f77_463c_a7bd_b12c146dd498.slice/crio-1a55374fa6d5c9ba264788459a86f5a1be813abeacd2af0c3e7a97231fde0bae WatchSource:0}: Error finding container 1a55374fa6d5c9ba264788459a86f5a1be813abeacd2af0c3e7a97231fde0bae: Status 404 returned error can't find the container with id 1a55374fa6d5c9ba264788459a86f5a1be813abeacd2af0c3e7a97231fde0bae
Sep 17 18:04:45.536782 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.536750    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:45.663617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.663596    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5"]
Sep 17 18:04:45.665397 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:45.665369    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cc5f99a_a7a9_4949_81f9_3572052afd7f.slice/crio-c0e4e7ebe2a2eed1c039d678fcb05ed6cfa1de2a9d4692cff390a991c6ce4051 WatchSource:0}: Error finding container c0e4e7ebe2a2eed1c039d678fcb05ed6cfa1de2a9d4692cff390a991c6ce4051: Status 404 returned error can't find the container with id c0e4e7ebe2a2eed1c039d678fcb05ed6cfa1de2a9d4692cff390a991c6ce4051
Sep 17 18:04:45.777596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.777568    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:45.889257 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.889217    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:45.998462 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:45.998384    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:46.044742 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:46.044718    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:04:46.124121 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:46.124086    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5" event={"ID":"7cc5f99a-a7a9-4949-81f9-3572052afd7f","Type":"ContainerStarted","Data":"2cc9025f76611aca8d11b7eba2c47842ade96a721b8819fefc12823cef71da82"}
Sep 17 18:04:46.124275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:46.124159    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5" event={"ID":"7cc5f99a-a7a9-4949-81f9-3572052afd7f","Type":"ContainerStarted","Data":"0c250dc80864e70d5b686f1f4292e2c4b151dbd98b759ae871f3d9a7980c3377"}
Sep 17 18:04:46.124275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:46.124184    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5" event={"ID":"7cc5f99a-a7a9-4949-81f9-3572052afd7f","Type":"ContainerStarted","Data":"c0e4e7ebe2a2eed1c039d678fcb05ed6cfa1de2a9d4692cff390a991c6ce4051"}
Sep 17 18:04:46.125401 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:46.125372    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hqlpd" event={"ID":"46e3b407-8f77-463c-a7bd-b12c146dd498","Type":"ContainerStarted","Data":"1a55374fa6d5c9ba264788459a86f5a1be813abeacd2af0c3e7a97231fde0bae"}
Sep 17 18:04:46.134925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:46.134898    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:46.292608 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:46.292569    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:46.840134 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:46.840090    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:46.929392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:46.929361    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:47.131043 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:47.130941    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5" event={"ID":"7cc5f99a-a7a9-4949-81f9-3572052afd7f","Type":"ContainerStarted","Data":"6f968ec28de09adeff41cd61ae7cceae58d2761261653484af59e453a761d136"}
Sep 17 18:04:47.134967 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:47.134941    2589 generic.go:356] "Generic (PLEG): container finished" podID="46e3b407-8f77-463c-a7bd-b12c146dd498" containerID="9814bae5729fbbb59efd746e2088c8b3d21c84a44c6b9e8139fe95ee8eb8c5df" exitCode=0
Sep 17 18:04:47.135106 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:47.134978    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hqlpd" event={"ID":"46e3b407-8f77-463c-a7bd-b12c146dd498","Type":"ContainerDied","Data":"9814bae5729fbbb59efd746e2088c8b3d21c84a44c6b9e8139fe95ee8eb8c5df"}
Sep 17 18:04:47.157353 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:47.157302    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-hb4k5" podStartSLOduration=1.155661934 podStartE2EDuration="2.157281223s" totalImagesPullingTime="1.001619289s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:47.155434451 +0000 UTC m=+122.835120304" watchObservedRunningTime="2026-09-17 18:04:47.157281223 +0000 UTC m=+122.836967078"
Sep 17 18:04:47.286766 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:47.286737    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:47.365313 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:47.365284    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:47.925846 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:47.925804    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:48.023139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:48.023103    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:48.140600 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:48.140550    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hqlpd" event={"ID":"46e3b407-8f77-463c-a7bd-b12c146dd498","Type":"ContainerStarted","Data":"8f4b533c7ca52488d6b149d1a5edc335be2d945eee902594eb2e215e9c37477e"}
Sep 17 18:04:48.140600 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:48.140603    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hqlpd" event={"ID":"46e3b407-8f77-463c-a7bd-b12c146dd498","Type":"ContainerStarted","Data":"8e3f2af275cb60367800eb075e8586dafa99e2f2152962a550e333ba6eb84efb"}
Sep 17 18:04:48.174498 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:48.174466    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:48.189764 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:48.189664    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hqlpd" podStartSLOduration=1.8351187740000001 podStartE2EDuration="3.189651157s" totalImagesPullingTime="1.282083364s" totalInitContainerRuntime="72.449019ms" isStatefulPod=true podCreationTimestamp="2026-09-17 18:04:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:48.189414403 +0000 UTC m=+123.869100258" watchObservedRunningTime="2026-09-17 18:04:48.189651157 +0000 UTC m=+123.869337012"
Sep 17 18:04:48.530842 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:48.530803    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:49.750556 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:49.750522    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:49.938123 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:49.938085    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:50.196918 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:50.196886    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:50.932217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:50.932176    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:51.024676 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.024638    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:51.287176 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.287140    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:51.801613 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.801577    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-68fbd6b88-bclkp"]
Sep 17 18:04:51.803884 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.803868    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:51.814104 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.814083    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 18:04:51.814801 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.814784    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ppsr6\""
Sep 17 18:04:51.814856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.814800    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 18:04:51.814856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.814784    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 18:04:51.814856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.814828    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 18:04:51.815044 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.815028    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 18:04:51.815240 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.815226    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 18:04:51.815240 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.815234    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 18:04:51.823963 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.823945    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 18:04:51.834363 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.834337    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68fbd6b88-bclkp"]
Sep 17 18:04:51.987490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.987453    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-oauth-config\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:51.987490 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.987490    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wv7ps\" (UniqueName: \"kubernetes.io/projected/b83154d9-7620-4894-980e-91bfd0b02a9c-kube-api-access-wv7ps\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:51.987925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.987518    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-service-ca\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:51.987925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.987565    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-console-config\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:51.987925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.987580    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-oauth-serving-cert\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:51.987925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.987611    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-trusted-ca-bundle\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:51.987925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:51.987683    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-serving-cert\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.088378 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.088304    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-service-ca\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.088378 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.088366    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-console-config\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.088556 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.088393    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-oauth-serving-cert\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.088556 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.088428    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-trusted-ca-bundle\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.088556 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.088479    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-serving-cert\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.088556 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.088507    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-oauth-config\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.088556 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.088534    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wv7ps\" (UniqueName: \"kubernetes.io/projected/b83154d9-7620-4894-980e-91bfd0b02a9c-kube-api-access-wv7ps\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.089148 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.089125    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-console-config\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.089432 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.089168    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-service-ca\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.089432 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.089201    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-oauth-serving-cert\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.089567 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.089547    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-trusted-ca-bundle\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.091122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.091102    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-serving-cert\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.091215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.091139    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-oauth-config\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.099265 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.099244    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wv7ps\" (UniqueName: \"kubernetes.io/projected/b83154d9-7620-4894-980e-91bfd0b02a9c-kube-api-access-wv7ps\") pod \"console-68fbd6b88-bclkp\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") " pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.113194 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.113174    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:04:52.391501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:52.391304    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-68fbd6b88-bclkp"]
Sep 17 18:04:52.394067 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:52.394043    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb83154d9_7620_4894_980e_91bfd0b02a9c.slice/crio-5a672285d608f259dc25140747c4ef3a7fe942e34036eed841ffa4d29569321a WatchSource:0}: Error finding container 5a672285d608f259dc25140747c4ef3a7fe942e34036eed841ffa4d29569321a: Status 404 returned error can't find the container with id 5a672285d608f259dc25140747c4ef3a7fe942e34036eed841ffa4d29569321a
Sep 17 18:04:53.161269 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:53.161221    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68fbd6b88-bclkp" event={"ID":"b83154d9-7620-4894-980e-91bfd0b02a9c","Type":"ContainerStarted","Data":"5a672285d608f259dc25140747c4ef3a7fe942e34036eed841ffa4d29569321a"}
Sep 17 18:04:54.882301 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:54.882271    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:54.973074 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:54.972992    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:56.174841 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.174817    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68fbd6b88-bclkp" event={"ID":"b83154d9-7620-4894-980e-91bfd0b02a9c","Type":"ContainerStarted","Data":"56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465"}
Sep 17 18:04:56.222681 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.220425    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-68fbd6b88-bclkp" podStartSLOduration=1.503953689 podStartE2EDuration="5.22040462s" totalImagesPullingTime="3.716450931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:56.218698081 +0000 UTC m=+131.898383937" watchObservedRunningTime="2026-09-17 18:04:56.22040462 +0000 UTC m=+131.900090477"
Sep 17 18:04:56.338155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.338120    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68fbd6b88-bclkp"]
Sep 17 18:04:56.351811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.351773    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:56.420333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.420245    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:04:56.422652 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.422633    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d310680-2035-4659-839c-2a1357a32aba-metrics-certs\") pod \"network-metrics-daemon-jmfh6\" (UID: \"3d310680-2035-4659-839c-2a1357a32aba\") " pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:04:56.456388 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.456357    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:56.500726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.500704    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qc24z\""
Sep 17 18:04:56.508949 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.508931    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-jmfh6"
Sep 17 18:04:56.778475 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:56.778445    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-jmfh6"]
Sep 17 18:04:56.781324 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:04:56.781300    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d310680_2035_4659_839c_2a1357a32aba.slice/crio-8b3eae1e543d05a10f1c3d4e1ceebe14fe9dde7f3a5ddcc6e2deb8efb114209c WatchSource:0}: Error finding container 8b3eae1e543d05a10f1c3d4e1ceebe14fe9dde7f3a5ddcc6e2deb8efb114209c: Status 404 returned error can't find the container with id 8b3eae1e543d05a10f1c3d4e1ceebe14fe9dde7f3a5ddcc6e2deb8efb114209c
Sep 17 18:04:57.179118 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:57.179038    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jmfh6" event={"ID":"3d310680-2035-4659-839c-2a1357a32aba","Type":"ContainerStarted","Data":"8b3eae1e543d05a10f1c3d4e1ceebe14fe9dde7f3a5ddcc6e2deb8efb114209c"}
Sep 17 18:04:58.184830 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:58.184789    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jmfh6" event={"ID":"3d310680-2035-4659-839c-2a1357a32aba","Type":"ContainerStarted","Data":"85ad34e494b993eec8c7a2e067740e22242a8bce0919a9b3816a01961236e099"}
Sep 17 18:04:58.185328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:58.184852    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-jmfh6" event={"ID":"3d310680-2035-4659-839c-2a1357a32aba","Type":"ContainerStarted","Data":"5880cfa352273df31128b3c523070211c157382ce3a93cb8c6101f5347d48c7a"}
Sep 17 18:04:58.218679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:58.218634    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-jmfh6" podStartSLOduration=130.269875555 podStartE2EDuration="2m11.218618465s" totalImagesPullingTime="948.74291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:04:58.217599103 +0000 UTC m=+133.897284958" watchObservedRunningTime="2026-09-17 18:04:58.218618465 +0000 UTC m=+133.898304318"
Sep 17 18:04:58.427878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:58.427843    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:58.527812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:58.527752    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:58.948375 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:58.948289    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:04:59.024462 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:04:59.024425    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:00.045051 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:00.045018    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"]
Sep 17 18:05:00.159881 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:00.159843    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:00.281296 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:00.281244    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:00.363580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:00.363478    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:02.114063 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:02.114017    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:05:07.451804 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:07.451771    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:07.629864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:07.629830    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:07.781591 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:07.781555    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:08.664994 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:08.664959    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:08.774965 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:08.774929    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:09.487331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:09.487297    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:09.765275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:09.765227    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:09.993753 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:09.993711    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:14.196953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:14.196918    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:14.289811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:14.289774    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:22.140653 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:22.140614    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:22.241521 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:22.241489    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:22.327357 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:22.327309    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:23.203905 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.203869    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-68fbd6b88-bclkp" podUID="b83154d9-7620-4894-980e-91bfd0b02a9c" containerName="console" containerID="cri-o://56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465" gracePeriod=15
Sep 17 18:05:23.335581 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.335547    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:23.414222 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.414190    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:23.448397 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.448368    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:05:23.522308 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522285    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-oauth-config\") pod \"b83154d9-7620-4894-980e-91bfd0b02a9c\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") "
Sep 17 18:05:23.522455 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522313    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wv7ps\" (UniqueName: \"kubernetes.io/projected/b83154d9-7620-4894-980e-91bfd0b02a9c-kube-api-access-wv7ps\") pod \"b83154d9-7620-4894-980e-91bfd0b02a9c\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") "
Sep 17 18:05:23.522455 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522335    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-console-config\") pod \"b83154d9-7620-4894-980e-91bfd0b02a9c\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") "
Sep 17 18:05:23.522455 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522366    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-trusted-ca-bundle\") pod \"b83154d9-7620-4894-980e-91bfd0b02a9c\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") "
Sep 17 18:05:23.522455 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522429    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-service-ca\") pod \"b83154d9-7620-4894-980e-91bfd0b02a9c\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") "
Sep 17 18:05:23.522665 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522468    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-oauth-serving-cert\") pod \"b83154d9-7620-4894-980e-91bfd0b02a9c\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") "
Sep 17 18:05:23.522665 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522499    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-serving-cert\") pod \"b83154d9-7620-4894-980e-91bfd0b02a9c\" (UID: \"b83154d9-7620-4894-980e-91bfd0b02a9c\") "
Sep 17 18:05:23.522848 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522803    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-console-config" (OuterVolumeSpecName: "console-config") pod "b83154d9-7620-4894-980e-91bfd0b02a9c" (UID: "b83154d9-7620-4894-980e-91bfd0b02a9c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:23.522926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522842    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-service-ca" (OuterVolumeSpecName: "service-ca") pod "b83154d9-7620-4894-980e-91bfd0b02a9c" (UID: "b83154d9-7620-4894-980e-91bfd0b02a9c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:23.522926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522867    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b83154d9-7620-4894-980e-91bfd0b02a9c" (UID: "b83154d9-7620-4894-980e-91bfd0b02a9c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:23.522926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.522879    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b83154d9-7620-4894-980e-91bfd0b02a9c" (UID: "b83154d9-7620-4894-980e-91bfd0b02a9c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:23.524789 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.524758    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b83154d9-7620-4894-980e-91bfd0b02a9c" (UID: "b83154d9-7620-4894-980e-91bfd0b02a9c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:23.524875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.524790    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b83154d9-7620-4894-980e-91bfd0b02a9c" (UID: "b83154d9-7620-4894-980e-91bfd0b02a9c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:23.524875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.524765    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b83154d9-7620-4894-980e-91bfd0b02a9c-kube-api-access-wv7ps" (OuterVolumeSpecName: "kube-api-access-wv7ps") pod "b83154d9-7620-4894-980e-91bfd0b02a9c" (UID: "b83154d9-7620-4894-980e-91bfd0b02a9c"). InnerVolumeSpecName "kube-api-access-wv7ps". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:23.622966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.622939    2589 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-console-config\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:23.622966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.622959    2589 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-trusted-ca-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:23.622966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.622968    2589 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:23.623145 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.622977    2589 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b83154d9-7620-4894-980e-91bfd0b02a9c-oauth-serving-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:23.623145 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.622985    2589 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-serving-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:23.623145 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.622994    2589 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b83154d9-7620-4894-980e-91bfd0b02a9c-console-oauth-config\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:23.623145 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:23.623001    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wv7ps\" (UniqueName: \"kubernetes.io/projected/b83154d9-7620-4894-980e-91bfd0b02a9c-kube-api-access-wv7ps\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:24.280170 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.280132    2589 generic.go:356] "Generic (PLEG): container finished" podID="b83154d9-7620-4894-980e-91bfd0b02a9c" containerID="56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465" exitCode=0
Sep 17 18:05:24.280170 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.280174    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-68fbd6b88-bclkp"
Sep 17 18:05:24.280648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.280185    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68fbd6b88-bclkp" event={"ID":"b83154d9-7620-4894-980e-91bfd0b02a9c","Type":"ContainerDied","Data":"56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465"}
Sep 17 18:05:24.280648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.280235    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-68fbd6b88-bclkp" event={"ID":"b83154d9-7620-4894-980e-91bfd0b02a9c","Type":"ContainerDied","Data":"5a672285d608f259dc25140747c4ef3a7fe942e34036eed841ffa4d29569321a"}
Sep 17 18:05:24.280648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.280267    2589 scope.go:117] "RemoveContainer" containerID="56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465"
Sep 17 18:05:24.289756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.289739    2589 scope.go:117] "RemoveContainer" containerID="56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465"
Sep 17 18:05:24.290062 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:05:24.290041    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465\": container with ID starting with 56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465 not found: ID does not exist" containerID="56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465"
Sep 17 18:05:24.290133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.290071    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465"} err="failed to get container status \"56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465\": rpc error: code = NotFound desc = could not find container \"56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465\": container with ID starting with 56ea0a78263f31d06e390519616f7aec87ee1788555adff7945d958ccb7e0465 not found: ID does not exist"
Sep 17 18:05:24.310857 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.310831    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-68fbd6b88-bclkp"]
Sep 17 18:05:24.316082 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.316061    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-68fbd6b88-bclkp"]
Sep 17 18:05:24.682284 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:24.682208    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b83154d9-7620-4894-980e-91bfd0b02a9c" path="/var/lib/kubelet/pods/b83154d9-7620-4894-980e-91bfd0b02a9c/volumes"
Sep 17 18:05:25.064540 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.064494    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf" podUID="cd7c076c-56fd-4f5e-890e-13a07d2f6d51" containerName="registry" containerID="cri-o://79f67b5ae3410b146e5c823779c1eb2f91b94df528b427a2020accbf5f36bc90" gracePeriod=30
Sep 17 18:05:25.285296 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.285257    2589 generic.go:356] "Generic (PLEG): container finished" podID="cd7c076c-56fd-4f5e-890e-13a07d2f6d51" containerID="79f67b5ae3410b146e5c823779c1eb2f91b94df528b427a2020accbf5f36bc90" exitCode=0
Sep 17 18:05:25.285657 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.285324    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf" event={"ID":"cd7c076c-56fd-4f5e-890e-13a07d2f6d51","Type":"ContainerDied","Data":"79f67b5ae3410b146e5c823779c1eb2f91b94df528b427a2020accbf5f36bc90"}
Sep 17 18:05:25.315190 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.315145    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:05:25.434578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.434548    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-trusted-ca\") pod \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") "
Sep 17 18:05:25.434712 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.434594    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdfv8\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-kube-api-access-kdfv8\") pod \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") "
Sep 17 18:05:25.434712 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.434613    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-bound-sa-token\") pod \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") "
Sep 17 18:05:25.434810 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.434793    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-image-registry-private-configuration\") pod \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") "
Sep 17 18:05:25.434846 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.434828    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-ca-trust-extracted\") pod \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") "
Sep 17 18:05:25.434891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.434854    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-certificates\") pod \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") "
Sep 17 18:05:25.434891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.434885    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") pod \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") "
Sep 17 18:05:25.434962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.434923    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-installation-pull-secrets\") pod \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\" (UID: \"cd7c076c-56fd-4f5e-890e-13a07d2f6d51\") "
Sep 17 18:05:25.434962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.434949    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "cd7c076c-56fd-4f5e-890e-13a07d2f6d51" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:25.435149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.435131    2589 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-trusted-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:25.435352 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.435313    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "cd7c076c-56fd-4f5e-890e-13a07d2f6d51" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:05:25.437436 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.437401    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "cd7c076c-56fd-4f5e-890e-13a07d2f6d51" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:25.437546 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.437502    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "cd7c076c-56fd-4f5e-890e-13a07d2f6d51" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:25.437622 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.437605    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-kube-api-access-kdfv8" (OuterVolumeSpecName: "kube-api-access-kdfv8") pod "cd7c076c-56fd-4f5e-890e-13a07d2f6d51" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51"). InnerVolumeSpecName "kube-api-access-kdfv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:05:25.437685 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.437662    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "cd7c076c-56fd-4f5e-890e-13a07d2f6d51" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:25.437912 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.437892    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "cd7c076c-56fd-4f5e-890e-13a07d2f6d51" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:05:25.443358 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.443337    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "cd7c076c-56fd-4f5e-890e-13a07d2f6d51" (UID: "cd7c076c-56fd-4f5e-890e-13a07d2f6d51"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:05:25.536318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.536243    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdfv8\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-kube-api-access-kdfv8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:25.536318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.536274    2589 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-bound-sa-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:25.536318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.536289    2589 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-image-registry-private-configuration\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:25.536318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.536304    2589 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-ca-trust-extracted\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:25.536318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.536317    2589 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-certificates\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:25.536616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.536329    2589 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-registry-tls\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:25.536616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:25.536341    2589 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/cd7c076c-56fd-4f5e-890e-13a07d2f6d51-installation-pull-secrets\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:05:26.291255 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:26.291230    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"
Sep 17 18:05:26.291697 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:26.291233    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7dd97fdbb8-6xzsf" event={"ID":"cd7c076c-56fd-4f5e-890e-13a07d2f6d51","Type":"ContainerDied","Data":"6b0caf653957c6de5f1aceee8290d0f06e7281a2800433720c77c8475ebbc300"}
Sep 17 18:05:26.291697 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:26.291375    2589 scope.go:117] "RemoveContainer" containerID="79f67b5ae3410b146e5c823779c1eb2f91b94df528b427a2020accbf5f36bc90"
Sep 17 18:05:26.293133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:26.293106    2589 generic.go:356] "Generic (PLEG): container finished" podID="7a7691fb-f931-4285-bce6-68555253947f" containerID="5263099a00aa526251130537273762caf58c9350c8cb208ac86ec8bcea5f76de" exitCode=0
Sep 17 18:05:26.293256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:26.293202    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2" event={"ID":"7a7691fb-f931-4285-bce6-68555253947f","Type":"ContainerDied","Data":"5263099a00aa526251130537273762caf58c9350c8cb208ac86ec8bcea5f76de"}
Sep 17 18:05:26.293589 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:26.293570    2589 scope.go:117] "RemoveContainer" containerID="5263099a00aa526251130537273762caf58c9350c8cb208ac86ec8bcea5f76de"
Sep 17 18:05:26.335911 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:26.335818    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"]
Sep 17 18:05:26.343748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:26.343701    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7dd97fdbb8-6xzsf"]
Sep 17 18:05:26.683742 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:26.683651    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd7c076c-56fd-4f5e-890e-13a07d2f6d51" path="/var/lib/kubelet/pods/cd7c076c-56fd-4f5e-890e-13a07d2f6d51/volumes"
Sep 17 18:05:27.060827 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:27.060772    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:27.300794 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:27.300760    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-57cdc4b768-6rgf2" event={"ID":"7a7691fb-f931-4285-bce6-68555253947f","Type":"ContainerStarted","Data":"66db12edadf7c9a0161d6559c8e13dc5e86d1d6834451c4c345281a3912eb85d"}
Sep 17 18:05:29.145315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:29.145278    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:05:29.246757 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:05:29.246730    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:01.117596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:01.117567    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:01.226780 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:01.226746    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:03.373714 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:03.373667    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:12.998866 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:12.998816    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:13.086487 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:13.086448    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:20.591822 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.591777    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9445698fd-8gqhf"]
Sep 17 18:06:20.592413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.592172    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd7c076c-56fd-4f5e-890e-13a07d2f6d51" containerName="registry"
Sep 17 18:06:20.592413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.592192    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd7c076c-56fd-4f5e-890e-13a07d2f6d51" containerName="registry"
Sep 17 18:06:20.592413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.592220    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b83154d9-7620-4894-980e-91bfd0b02a9c" containerName="console"
Sep 17 18:06:20.592413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.592229    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="b83154d9-7620-4894-980e-91bfd0b02a9c" containerName="console"
Sep 17 18:06:20.592413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.592295    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="b83154d9-7620-4894-980e-91bfd0b02a9c" containerName="console"
Sep 17 18:06:20.592413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.592307    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd7c076c-56fd-4f5e-890e-13a07d2f6d51" containerName="registry"
Sep 17 18:06:20.594066 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.594044    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.597412 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.597391    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 17 18:06:20.598103 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.598085    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-ppsr6\""
Sep 17 18:06:20.599118 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.599099    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 17 18:06:20.599212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.599116    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 17 18:06:20.599279 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.599248    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 17 18:06:20.599367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.599338    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 17 18:06:20.599715 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.599696    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 17 18:06:20.599798 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.599696    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 17 18:06:20.605051 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.605035    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 17 18:06:20.611451 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.611430    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9445698fd-8gqhf"]
Sep 17 18:06:20.671726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.671690    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:20.732260 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.732235    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-trusted-ca-bundle\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.732392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.732285    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-oauth-config\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.732392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.732313    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-oauth-serving-cert\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.732392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.732353    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-service-ca\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.732392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.732370    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdcj7\" (UniqueName: \"kubernetes.io/projected/26f837ae-03db-494a-90d3-1afc2991651d-kube-api-access-wdcj7\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.732522 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.732395    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-serving-cert\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.732522 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.732418    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-console-config\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.771861 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.771835    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:20.833349 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.833309    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-oauth-config\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.833349 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.833361    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-oauth-serving-cert\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.833530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.833381    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-service-ca\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.833530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.833396    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdcj7\" (UniqueName: \"kubernetes.io/projected/26f837ae-03db-494a-90d3-1afc2991651d-kube-api-access-wdcj7\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.833530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.833426    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-serving-cert\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.833530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.833449    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-console-config\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.833530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.833467    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-trusted-ca-bundle\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.834283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.834134    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-oauth-serving-cert\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.834283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.834182    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-service-ca\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.834283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.834236    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-console-config\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.834283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.834267    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-trusted-ca-bundle\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.836024 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.835990    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-oauth-config\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.836119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.836101    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-serving-cert\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.843597 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.843538    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdcj7\" (UniqueName: \"kubernetes.io/projected/26f837ae-03db-494a-90d3-1afc2991651d-kube-api-access-wdcj7\") pod \"console-9445698fd-8gqhf\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") " pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:20.903790 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:20.903768    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:21.180775 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:21.180704    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9445698fd-8gqhf"]
Sep 17 18:06:21.182946 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:06:21.182923    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f837ae_03db_494a_90d3_1afc2991651d.slice/crio-6b4658460670eb97bd5969a40ffc0ba13dcf004895d298cc34aa3a20c11c7302 WatchSource:0}: Error finding container 6b4658460670eb97bd5969a40ffc0ba13dcf004895d298cc34aa3a20c11c7302: Status 404 returned error can't find the container with id 6b4658460670eb97bd5969a40ffc0ba13dcf004895d298cc34aa3a20c11c7302
Sep 17 18:06:21.489541 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:21.489462    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9445698fd-8gqhf" event={"ID":"26f837ae-03db-494a-90d3-1afc2991651d","Type":"ContainerStarted","Data":"6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9"}
Sep 17 18:06:21.489541 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:21.489511    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9445698fd-8gqhf" event={"ID":"26f837ae-03db-494a-90d3-1afc2991651d","Type":"ContainerStarted","Data":"6b4658460670eb97bd5969a40ffc0ba13dcf004895d298cc34aa3a20c11c7302"}
Sep 17 18:06:21.516084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:21.516030    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9445698fd-8gqhf" podStartSLOduration=1.515996989 podStartE2EDuration="1.515996989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:06:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:06:21.514715137 +0000 UTC m=+217.194400991" watchObservedRunningTime="2026-09-17 18:06:21.515996989 +0000 UTC m=+217.195682843"
Sep 17 18:06:24.745518 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:24.745481    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:29.227964 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:29.227931    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:29.323909 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:29.323871    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:30.903912 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:30.903879    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:30.904331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:30.903931    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:30.908739 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:30.908718    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:31.527845 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:31.527816    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:06:31.741065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:31.741036    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:31.839062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:31.838944    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:31.926926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:31.926894    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:46.338178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:46.338123    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:46.410136 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:46.410106    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:52.060858 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:52.060822    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:52.172806 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:52.172776    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:06:59.927685 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:06:59.927609    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:00.021257 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:00.021217    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:27.312962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:27.312918    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:28.227870 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:28.227824    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:30.920460 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:30.920427    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:31.020802 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:31.020773    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:32.135817 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.135779    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:32.247452 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.247419    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:32.489506 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.489407    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:32.633521 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.633485    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:32.816863 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.816824    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-px2x6"]
Sep 17 18:07:32.819978 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.819963    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:32.822756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.822725    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 17 18:07:32.823697 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.823678    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 17 18:07:32.823798 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.823678    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hv4pj\""
Sep 17 18:07:32.840983 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.840956    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-px2x6"]
Sep 17 18:07:32.956328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.956304    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4tnn\" (UniqueName: \"kubernetes.io/projected/2da58800-26a8-4dc9-a92b-e4b32f617c75-kube-api-access-p4tnn\") pod \"cert-manager-webhook-6486ffd997-px2x6\" (UID: \"2da58800-26a8-4dc9-a92b-e4b32f617c75\") " pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:32.956440 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:32.956345    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2da58800-26a8-4dc9-a92b-e4b32f617c75-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-px2x6\" (UID: \"2da58800-26a8-4dc9-a92b-e4b32f617c75\") " pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:33.056869 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:33.056827    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2da58800-26a8-4dc9-a92b-e4b32f617c75-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-px2x6\" (UID: \"2da58800-26a8-4dc9-a92b-e4b32f617c75\") " pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:33.057046 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:33.056902    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4tnn\" (UniqueName: \"kubernetes.io/projected/2da58800-26a8-4dc9-a92b-e4b32f617c75-kube-api-access-p4tnn\") pod \"cert-manager-webhook-6486ffd997-px2x6\" (UID: \"2da58800-26a8-4dc9-a92b-e4b32f617c75\") " pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:33.070470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:33.070400    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2da58800-26a8-4dc9-a92b-e4b32f617c75-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-px2x6\" (UID: \"2da58800-26a8-4dc9-a92b-e4b32f617c75\") " pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:33.070577 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:33.070556    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4tnn\" (UniqueName: \"kubernetes.io/projected/2da58800-26a8-4dc9-a92b-e4b32f617c75-kube-api-access-p4tnn\") pod \"cert-manager-webhook-6486ffd997-px2x6\" (UID: \"2da58800-26a8-4dc9-a92b-e4b32f617c75\") " pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:33.129438 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:33.129403    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:33.400946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:33.400923    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-px2x6"]
Sep 17 18:07:33.403722 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:07:33.403693    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2da58800_26a8_4dc9_a92b_e4b32f617c75.slice/crio-8ce5901a360cb8a122e1c985c78fcbe6bc36b3256ac2a4d61d8aeb2491d63dcd WatchSource:0}: Error finding container 8ce5901a360cb8a122e1c985c78fcbe6bc36b3256ac2a4d61d8aeb2491d63dcd: Status 404 returned error can't find the container with id 8ce5901a360cb8a122e1c985c78fcbe6bc36b3256ac2a4d61d8aeb2491d63dcd
Sep 17 18:07:33.734542 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:33.734458    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6" event={"ID":"2da58800-26a8-4dc9-a92b-e4b32f617c75","Type":"ContainerStarted","Data":"8ce5901a360cb8a122e1c985c78fcbe6bc36b3256ac2a4d61d8aeb2491d63dcd"}
Sep 17 18:07:34.802326 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:34.802296    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:34.993178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:34.993139    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:36.376349 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:36.376280    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:37.751214 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:37.751176    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6" event={"ID":"2da58800-26a8-4dc9-a92b-e4b32f617c75","Type":"ContainerStarted","Data":"8d1662b89ac4b3dc48f67a1780231825955082db7e0dcb0521f9cfdb60e4442f"}
Sep 17 18:07:37.751614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:37.751253    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:37.778784 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:37.778733    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6" podStartSLOduration=1.9896048400000002 podStartE2EDuration="5.778718459s" totalImagesPullingTime="3.789113619s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:07:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:07:37.777903962 +0000 UTC m=+293.457589816" watchObservedRunningTime="2026-09-17 18:07:37.778718459 +0000 UTC m=+293.458404313"
Sep 17 18:07:38.164912 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:38.164867    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:38.274583 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:38.274548    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:41.386919 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:41.386887    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:41.518908 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:41.518881    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:42.397421 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:42.397392    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:43.757229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:43.757195    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-px2x6"
Sep 17 18:07:45.256734 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:45.256682    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:45.373862 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:45.373832    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:46.542489 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:46.542452    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:07:46.546374 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:46.546354    2589 kubelet.go:1648] "Image garbage collection succeeded"
Sep 17 18:07:46.548793 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:46.548769    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:07:47.748118 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:47.748082    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:47.845229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:47.845198    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:51.488345 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:51.488303    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:51.591271 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:51.591233    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:07:53.161984 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:07:53.161953    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:06.053847 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:06.053784    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:06.204859 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:06.204811    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:06.616123 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:06.616089    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:07.004817 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.004787    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:07.105517 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.105491    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:07.191525 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.191489    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"]
Sep 17 18:08:07.193699 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.193682    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.196941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.196915    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 17 18:08:07.197095 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.196960    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 17 18:08:07.197249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.197230    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 17 18:08:07.197454 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.197441    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 17 18:08:07.197496 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.197480    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-8977h\""
Sep 17 18:08:07.215898 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.215876    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"]
Sep 17 18:08:07.289837 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.289746    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:07.296743 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.296717    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97efffa8-eaa8-467a-afe1-a1ac0a849718-tmp\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.296872 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.296767    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97efffa8-eaa8-467a-afe1-a1ac0a849718-apiservice-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.296872 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.296849    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr5jp\" (UniqueName: \"kubernetes.io/projected/97efffa8-eaa8-467a-afe1-a1ac0a849718-kube-api-access-gr5jp\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.296948 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.296928    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97efffa8-eaa8-467a-afe1-a1ac0a849718-manifests\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.296988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.296960    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97efffa8-eaa8-467a-afe1-a1ac0a849718-webhook-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.393884 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.393854    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:07.397842 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.397822    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97efffa8-eaa8-467a-afe1-a1ac0a849718-apiservice-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.397929 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.397866    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr5jp\" (UniqueName: \"kubernetes.io/projected/97efffa8-eaa8-467a-afe1-a1ac0a849718-kube-api-access-gr5jp\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.397929 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.397899    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97efffa8-eaa8-467a-afe1-a1ac0a849718-manifests\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.397929 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.397918    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97efffa8-eaa8-467a-afe1-a1ac0a849718-webhook-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.398110 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.397939    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97efffa8-eaa8-467a-afe1-a1ac0a849718-tmp\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.398387 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.398357    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/97efffa8-eaa8-467a-afe1-a1ac0a849718-manifests\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.400417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.400387    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/97efffa8-eaa8-467a-afe1-a1ac0a849718-tmp\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.400566 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.400552    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/97efffa8-eaa8-467a-afe1-a1ac0a849718-webhook-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.400702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.400683    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/97efffa8-eaa8-467a-afe1-a1ac0a849718-apiservice-cert\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.411519 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.411494    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr5jp\" (UniqueName: \"kubernetes.io/projected/97efffa8-eaa8-467a-afe1-a1ac0a849718-kube-api-access-gr5jp\") pod \"opendatahub-operator-controller-manager-648c847c44-hrrfd\" (UID: \"97efffa8-eaa8-467a-afe1-a1ac0a849718\") " pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.503905 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.503872    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:07.519465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.519427    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:07.709641 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.709603    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:07.813299 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.813222    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:07.828088 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.826935    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"]
Sep 17 18:08:07.830428 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:08:07.830400    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97efffa8_eaa8_467a_afe1_a1ac0a849718.slice/crio-9b6b8987667b1558d6217e16e7cc1528efa7a8fc82a6295951d7dea947daece7 WatchSource:0}: Error finding container 9b6b8987667b1558d6217e16e7cc1528efa7a8fc82a6295951d7dea947daece7: Status 404 returned error can't find the container with id 9b6b8987667b1558d6217e16e7cc1528efa7a8fc82a6295951d7dea947daece7
Sep 17 18:08:07.831201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.831185    2589 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:08:07.853336 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.853308    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd" event={"ID":"97efffa8-eaa8-467a-afe1-a1ac0a849718","Type":"ContainerStarted","Data":"9b6b8987667b1558d6217e16e7cc1528efa7a8fc82a6295951d7dea947daece7"}
Sep 17 18:08:07.912588 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:07.912553    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:08.013454 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:08.013426    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:08.149046 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:08.148935    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:11.565720 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:11.565684    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:11.665658 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:11.665622    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:11.869265 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:11.869181    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd" event={"ID":"97efffa8-eaa8-467a-afe1-a1ac0a849718","Type":"ContainerStarted","Data":"3f965c0c6e8615775a1145eef6877fc2ff23cc06b502cb13ab39f2ea78053793"}
Sep 17 18:08:12.874357 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:12.874329    2589 generic.go:356] "Generic (PLEG): container finished" podID="97efffa8-eaa8-467a-afe1-a1ac0a849718" containerID="3f965c0c6e8615775a1145eef6877fc2ff23cc06b502cb13ab39f2ea78053793" exitCode=0
Sep 17 18:08:12.874720 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:12.874363    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd" event={"ID":"97efffa8-eaa8-467a-afe1-a1ac0a849718","Type":"ContainerDied","Data":"3f965c0c6e8615775a1145eef6877fc2ff23cc06b502cb13ab39f2ea78053793"}
Sep 17 18:08:13.880242 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:13.880210    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd" event={"ID":"97efffa8-eaa8-467a-afe1-a1ac0a849718","Type":"ContainerStarted","Data":"a4793317031133cef4ee5bf99c1d5b07ceb051361da1c97267b834a01405c64c"}
Sep 17 18:08:13.880632 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:13.880356    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:13.916160 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:13.916108    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd" podStartSLOduration=2.392872439 podStartE2EDuration="6.916093839s" totalImagesPullingTime="3.468726302s" totalInitContainerRuntime="1.054495098s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:13.915098429 +0000 UTC m=+329.594784282" watchObservedRunningTime="2026-09-17 18:08:13.916093839 +0000 UTC m=+329.595779693"
Sep 17 18:08:24.886132 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:24.886103    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-648c847c44-hrrfd"
Sep 17 18:08:25.939290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:25.939215    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"]
Sep 17 18:08:25.941348 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:25.941330    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:25.944040 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:25.943994    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 17 18:08:25.945093 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:25.945075    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-tmvdk\""
Sep 17 18:08:25.947129 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:25.947093    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 17 18:08:25.947239 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:25.947128    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 17 18:08:25.947304 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:25.947128    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 17 18:08:25.949000 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:25.948983    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 17 18:08:25.970477 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:25.970452    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"]
Sep 17 18:08:26.033197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.033163    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9860d101-be11-48fa-a3fc-061808d4ef2f-metrics-cert\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.033317 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.033215    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9860d101-be11-48fa-a3fc-061808d4ef2f-manager-config\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.033317 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.033286    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9860d101-be11-48fa-a3fc-061808d4ef2f-cert\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.033405 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.033319    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7js7b\" (UniqueName: \"kubernetes.io/projected/9860d101-be11-48fa-a3fc-061808d4ef2f-kube-api-access-7js7b\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.134186 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.134155    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9860d101-be11-48fa-a3fc-061808d4ef2f-metrics-cert\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.134335 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.134206    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9860d101-be11-48fa-a3fc-061808d4ef2f-manager-config\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.134335 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.134255    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9860d101-be11-48fa-a3fc-061808d4ef2f-cert\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.134335 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.134284    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7js7b\" (UniqueName: \"kubernetes.io/projected/9860d101-be11-48fa-a3fc-061808d4ef2f-kube-api-access-7js7b\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.134965 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.134894    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/9860d101-be11-48fa-a3fc-061808d4ef2f-manager-config\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.136864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.136842    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/9860d101-be11-48fa-a3fc-061808d4ef2f-metrics-cert\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.137057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.137039    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9860d101-be11-48fa-a3fc-061808d4ef2f-cert\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.148662 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.148641    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7js7b\" (UniqueName: \"kubernetes.io/projected/9860d101-be11-48fa-a3fc-061808d4ef2f-kube-api-access-7js7b\") pod \"lws-controller-manager-b8b8795dc-jhzhb\" (UID: \"9860d101-be11-48fa-a3fc-061808d4ef2f\") " pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.250761 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.250690    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:26.534813 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.534786    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"]
Sep 17 18:08:26.537824 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:08:26.537800    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9860d101_be11_48fa_a3fc_061808d4ef2f.slice/crio-63eadc8ac32e6cddb6755b477a5f231d22fe0d320d2dea1d8775d3ad3a104ed6 WatchSource:0}: Error finding container 63eadc8ac32e6cddb6755b477a5f231d22fe0d320d2dea1d8775d3ad3a104ed6: Status 404 returned error can't find the container with id 63eadc8ac32e6cddb6755b477a5f231d22fe0d320d2dea1d8775d3ad3a104ed6
Sep 17 18:08:26.926832 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:26.926737    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb" event={"ID":"9860d101-be11-48fa-a3fc-061808d4ef2f","Type":"ContainerStarted","Data":"63eadc8ac32e6cddb6755b477a5f231d22fe0d320d2dea1d8775d3ad3a104ed6"}
Sep 17 18:08:27.775893 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:27.775850    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:28.375625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.375594    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"]
Sep 17 18:08:28.377583 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.377567    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:28.389112 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:08:28.389089    2589 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 17 18:08:28.389302 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:08:28.389280    2589 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-creds\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" type="*v1.Secret"
Sep 17 18:08:28.389434 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:08:28.389415    2589 status_manager.go:1018] "Failed to get status for pod" err="pods \"kube-auth-proxy-7bc8cbff6f-6vkds\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="a0f5e960-fac5-4865-a213-2419bf880c14" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:28.390383 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:08:28.390362    2589 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-dockercfg-ldmzf\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ldmzf\"" type="*v1.Secret"
Sep 17 18:08:28.391642 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:08:28.391617    2589 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kube-auth-proxy-tls\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret"
Sep 17 18:08:28.391642 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:08:28.391621    2589 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 17 18:08:28.395704 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.395685    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"]
Sep 17 18:08:28.453891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.453856    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0f5e960-fac5-4865-a213-2419bf880c14-tmp\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:28.454057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.453929    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjlg\" (UniqueName: \"kubernetes.io/projected/a0f5e960-fac5-4865-a213-2419bf880c14-kube-api-access-msjlg\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:28.454057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.453953    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f5e960-fac5-4865-a213-2419bf880c14-tls-certs\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:28.555187 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.555158    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0f5e960-fac5-4865-a213-2419bf880c14-tmp\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:28.555343 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.555211    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjlg\" (UniqueName: \"kubernetes.io/projected/a0f5e960-fac5-4865-a213-2419bf880c14-kube-api-access-msjlg\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:28.555387 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.555333    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f5e960-fac5-4865-a213-2419bf880c14-tls-certs\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:28.557466 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:28.557447    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a0f5e960-fac5-4865-a213-2419bf880c14-tmp\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:29.367238 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.367209    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 17 18:08:29.378974 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.378945    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0f5e960-fac5-4865-a213-2419bf880c14-tls-certs\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:29.456425 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.456379    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 17 18:08:29.574201 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:08:29.574168    2589 projected.go:291] Couldn't get configMap openshift-ingress/kube-root-ca.crt: failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:08:29.638762 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.638673    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:29.667127 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.667103    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 17 18:08:29.717386 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.717356    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-ldmzf\""
Sep 17 18:08:29.897069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.896961    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 17 18:08:29.904552 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:08:29.904530    2589 projected.go:196] Error preparing data for projected volume kube-api-access-msjlg for pod openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds: failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:08:29.904689 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:08:29.904606    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a0f5e960-fac5-4865-a213-2419bf880c14-kube-api-access-msjlg podName:a0f5e960-fac5-4865-a213-2419bf880c14 nodeName:}" failed. No retries permitted until 2026-09-17 18:08:30.404583427 +0000 UTC m=+346.084269273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-msjlg" (UniqueName: "kubernetes.io/projected/a0f5e960-fac5-4865-a213-2419bf880c14-kube-api-access-msjlg") pod "kube-auth-proxy-7bc8cbff6f-6vkds" (UID: "a0f5e960-fac5-4865-a213-2419bf880c14") : failed to sync configmap cache: timed out waiting for the condition
Sep 17 18:08:29.939833 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.939800    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb" event={"ID":"9860d101-be11-48fa-a3fc-061808d4ef2f","Type":"ContainerStarted","Data":"fe2afc2636e1049b61747b1c93b538798b15f51818de343014e4640d07cab809"}
Sep 17 18:08:29.939948 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.939921    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:29.992054 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:29.991979    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb" podStartSLOduration=1.822818436 podStartE2EDuration="4.991962648s" totalImagesPullingTime="3.169144212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:29.991233608 +0000 UTC m=+345.670919463" watchObservedRunningTime="2026-09-17 18:08:29.991962648 +0000 UTC m=+345.671648563"
Sep 17 18:08:30.471471 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:30.471424    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjlg\" (UniqueName: \"kubernetes.io/projected/a0f5e960-fac5-4865-a213-2419bf880c14-kube-api-access-msjlg\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:30.474161 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:30.474134    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjlg\" (UniqueName: \"kubernetes.io/projected/a0f5e960-fac5-4865-a213-2419bf880c14-kube-api-access-msjlg\") pod \"kube-auth-proxy-7bc8cbff6f-6vkds\" (UID: \"a0f5e960-fac5-4865-a213-2419bf880c14\") " pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:30.487058 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:30.487038    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"
Sep 17 18:08:30.693029 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:30.692987    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:30.782283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:30.782252    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds"]
Sep 17 18:08:30.785701 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:08:30.785676    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f5e960_fac5_4865_a213_2419bf880c14.slice/crio-cd6be23adda71fcfcf12af80dfaaefe09ec666fcc7f10519dc32769a7ca3c83d WatchSource:0}: Error finding container cd6be23adda71fcfcf12af80dfaaefe09ec666fcc7f10519dc32769a7ca3c83d: Status 404 returned error can't find the container with id cd6be23adda71fcfcf12af80dfaaefe09ec666fcc7f10519dc32769a7ca3c83d
Sep 17 18:08:30.907788 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:30.907752    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:30.944871 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:30.944837    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds" event={"ID":"a0f5e960-fac5-4865-a213-2419bf880c14","Type":"ContainerStarted","Data":"cd6be23adda71fcfcf12af80dfaaefe09ec666fcc7f10519dc32769a7ca3c83d"}
Sep 17 18:08:32.724562 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:32.724516    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:33.958684 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:33.958645    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds" event={"ID":"a0f5e960-fac5-4865-a213-2419bf880c14","Type":"ContainerStarted","Data":"ab390cb340dbb9280540395b223fa108944a20a2b6904552908ed9b29eaa2e1a"}
Sep 17 18:08:33.985799 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:33.985747    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bc8cbff6f-6vkds" podStartSLOduration=3.529161903 podStartE2EDuration="5.985732863s" totalImagesPullingTime="2.45657096s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:08:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:08:33.984863126 +0000 UTC m=+349.664548980" watchObservedRunningTime="2026-09-17 18:08:33.985732863 +0000 UTC m=+349.665418717"
Sep 17 18:08:35.279987 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:35.279950    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:35.398474 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:35.398435    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:36.346277 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:36.346242    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:36.446585 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:36.446554    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:36.562056 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:36.561987    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:37.556805 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:37.556768    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:37.649508 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:37.649475    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:38.831468 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:38.831428    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:38.940310 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:38.940274    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:39.032182 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:39.032148    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:39.246037 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:39.245928    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:39.465964 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:39.465929    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:39.635548 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:39.635517    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:40.947123 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:40.947096    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-b8b8795dc-jhzhb"
Sep 17 18:08:49.460099 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:49.460059    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:49.574176 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:49.574138    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:50.459899 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:50.459858    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:51.066361 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:51.066332    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:08:51.186724 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:08:51.186695    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:05.399772 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:05.399735    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:39.859479 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:39.859440    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:39.952953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:39.952920    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:46.357918 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:46.357885    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:46.444817 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:46.444781    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:53.578627 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:53.578591    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:54.366277 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:54.366240    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:55.154367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:55.154333    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:57.247578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:57.247490    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:57.340119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:57.340081    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:57.374325 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:57.374294    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9445698fd-8gqhf"]
Sep 17 18:09:57.543403 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:57.543375    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:57.641810 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:57.641783    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:09:58.619725 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:09:58.619688    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:00.161682 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:00.161648    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:00.417167 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:00.417083    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:00.638136 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:00.638104    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:02.787805 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:02.787772    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:02.885196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:02.885152    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:03.356875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:03.356835    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:04.143596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:04.143557    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:06.039022 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.038981    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"]
Sep 17 18:10:06.039459 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.039344    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:06.041186 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.041170    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.044351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.044329    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 17 18:10:06.044485 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.044450    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 17 18:10:06.044485 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.044461    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 17 18:10:06.044603 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.044524    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 17 18:10:06.045048 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.045029    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vb8rw\""
Sep 17 18:10:06.054283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.054261    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"]
Sep 17 18:10:06.208998 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.208967    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6539508f-a67f-4f99-92c5-9b8492b28616-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.209174 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.209063    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82ct9\" (UniqueName: \"kubernetes.io/projected/6539508f-a67f-4f99-92c5-9b8492b28616-kube-api-access-82ct9\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.209174 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.209104    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6539508f-a67f-4f99-92c5-9b8492b28616-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.310404 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.310318    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6539508f-a67f-4f99-92c5-9b8492b28616-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.310404 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.310363    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6539508f-a67f-4f99-92c5-9b8492b28616-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.310617 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:10:06.310489    2589 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 17 18:10:06.310617 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:10:06.310539    2589 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6539508f-a67f-4f99-92c5-9b8492b28616-plugin-serving-cert podName:6539508f-a67f-4f99-92c5-9b8492b28616 nodeName:}" failed. No retries permitted until 2026-09-17 18:10:06.8105245 +0000 UTC m=+442.490210331 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/6539508f-a67f-4f99-92c5-9b8492b28616-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-n2mnc" (UID: "6539508f-a67f-4f99-92c5-9b8492b28616") : secret "plugin-serving-cert" not found
Sep 17 18:10:06.310617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.310552    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82ct9\" (UniqueName: \"kubernetes.io/projected/6539508f-a67f-4f99-92c5-9b8492b28616-kube-api-access-82ct9\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.311061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.311037    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/6539508f-a67f-4f99-92c5-9b8492b28616-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.326650 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.326618    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82ct9\" (UniqueName: \"kubernetes.io/projected/6539508f-a67f-4f99-92c5-9b8492b28616-kube-api-access-82ct9\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.814777 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.814744    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6539508f-a67f-4f99-92c5-9b8492b28616-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.817286 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.817254    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/6539508f-a67f-4f99-92c5-9b8492b28616-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-n2mnc\" (UID: \"6539508f-a67f-4f99-92c5-9b8492b28616\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:06.869308 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.869284    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:06.950606 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:06.950582    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"
Sep 17 18:10:07.224343 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:07.224278    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc"]
Sep 17 18:10:07.226951 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:10:07.226923    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6539508f_a67f_4f99_92c5_9b8492b28616.slice/crio-17aa56aafbdc802fc389321af2b61365bf795d8c672363d7802b447746cefd06 WatchSource:0}: Error finding container 17aa56aafbdc802fc389321af2b61365bf795d8c672363d7802b447746cefd06: Status 404 returned error can't find the container with id 17aa56aafbdc802fc389321af2b61365bf795d8c672363d7802b447746cefd06
Sep 17 18:10:07.309699 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:07.309662    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc" event={"ID":"6539508f-a67f-4f99-92c5-9b8492b28616","Type":"ContainerStarted","Data":"17aa56aafbdc802fc389321af2b61365bf795d8c672363d7802b447746cefd06"}
Sep 17 18:10:08.390754 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:08.390667    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:08.499551 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:08.499515    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:08.766201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:08.766173    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:08.866060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:08.866020    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:09.161495 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:09.161393    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:09.255044 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:09.254957    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:12.336427 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:12.336390    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc" event={"ID":"6539508f-a67f-4f99-92c5-9b8492b28616","Type":"ContainerStarted","Data":"741328e88a7639c071c4915ca78ce85a9e9a10b59a27891bbeb21115c2dc088a"}
Sep 17 18:10:12.372145 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:12.372096    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-n2mnc" podStartSLOduration=1.9479853 podStartE2EDuration="6.372080778s" totalImagesPullingTime="4.424095478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:10:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:10:12.370785441 +0000 UTC m=+448.050471294" watchObservedRunningTime="2026-09-17 18:10:12.372080778 +0000 UTC m=+448.051766633"
Sep 17 18:10:15.434094 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:15.434059    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:15.559241 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:15.559172    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:20.207413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:20.207367    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:20.309985 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:20.309954    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:22.393391 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.393353    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9445698fd-8gqhf" podUID="26f837ae-03db-494a-90d3-1afc2991651d" containerName="console" containerID="cri-o://6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9" gracePeriod=15
Sep 17 18:10:22.629944 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.629923    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:10:22.674244 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.674169    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:22.736462 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.736430    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-oauth-serving-cert\") pod \"26f837ae-03db-494a-90d3-1afc2991651d\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") "
Sep 17 18:10:22.736462 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.736473    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-serving-cert\") pod \"26f837ae-03db-494a-90d3-1afc2991651d\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") "
Sep 17 18:10:22.736691 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.736503    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-service-ca\") pod \"26f837ae-03db-494a-90d3-1afc2991651d\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") "
Sep 17 18:10:22.736691 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.736537    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-trusted-ca-bundle\") pod \"26f837ae-03db-494a-90d3-1afc2991651d\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") "
Sep 17 18:10:22.736691 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.736590    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-oauth-config\") pod \"26f837ae-03db-494a-90d3-1afc2991651d\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") "
Sep 17 18:10:22.736691 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.736624    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-console-config\") pod \"26f837ae-03db-494a-90d3-1afc2991651d\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") "
Sep 17 18:10:22.736691 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.736661    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdcj7\" (UniqueName: \"kubernetes.io/projected/26f837ae-03db-494a-90d3-1afc2991651d-kube-api-access-wdcj7\") pod \"26f837ae-03db-494a-90d3-1afc2991651d\" (UID: \"26f837ae-03db-494a-90d3-1afc2991651d\") "
Sep 17 18:10:22.736987 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.736861    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "26f837ae-03db-494a-90d3-1afc2991651d" (UID: "26f837ae-03db-494a-90d3-1afc2991651d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:22.737097 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.737063    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-service-ca" (OuterVolumeSpecName: "service-ca") pod "26f837ae-03db-494a-90d3-1afc2991651d" (UID: "26f837ae-03db-494a-90d3-1afc2991651d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:22.737156 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.737131    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-console-config" (OuterVolumeSpecName: "console-config") pod "26f837ae-03db-494a-90d3-1afc2991651d" (UID: "26f837ae-03db-494a-90d3-1afc2991651d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:22.737259 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.737241    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "26f837ae-03db-494a-90d3-1afc2991651d" (UID: "26f837ae-03db-494a-90d3-1afc2991651d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:10:22.738895 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.738873    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "26f837ae-03db-494a-90d3-1afc2991651d" (UID: "26f837ae-03db-494a-90d3-1afc2991651d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:22.738967 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.738950    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "26f837ae-03db-494a-90d3-1afc2991651d" (UID: "26f837ae-03db-494a-90d3-1afc2991651d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:10:22.739263 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.739239    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f837ae-03db-494a-90d3-1afc2991651d-kube-api-access-wdcj7" (OuterVolumeSpecName: "kube-api-access-wdcj7") pod "26f837ae-03db-494a-90d3-1afc2991651d" (UID: "26f837ae-03db-494a-90d3-1afc2991651d"). InnerVolumeSpecName "kube-api-access-wdcj7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:10:22.774846 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.774810    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:22.837620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.837594    2589 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:10:22.837620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.837618    2589 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-trusted-ca-bundle\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:10:22.837803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.837629    2589 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-oauth-config\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:10:22.837803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.837638    2589 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-console-config\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:10:22.837803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.837648    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdcj7\" (UniqueName: \"kubernetes.io/projected/26f837ae-03db-494a-90d3-1afc2991651d-kube-api-access-wdcj7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:10:22.837803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.837656    2589 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/26f837ae-03db-494a-90d3-1afc2991651d-oauth-serving-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:10:22.837803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:22.837664    2589 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/26f837ae-03db-494a-90d3-1afc2991651d-console-serving-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:10:23.383625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:23.383574    2589 generic.go:356] "Generic (PLEG): container finished" podID="26f837ae-03db-494a-90d3-1afc2991651d" containerID="6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9" exitCode=0
Sep 17 18:10:23.383625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:23.383614    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9445698fd-8gqhf"
Sep 17 18:10:23.383893 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:23.383655    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9445698fd-8gqhf" event={"ID":"26f837ae-03db-494a-90d3-1afc2991651d","Type":"ContainerDied","Data":"6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9"}
Sep 17 18:10:23.383893 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:23.383704    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9445698fd-8gqhf" event={"ID":"26f837ae-03db-494a-90d3-1afc2991651d","Type":"ContainerDied","Data":"6b4658460670eb97bd5969a40ffc0ba13dcf004895d298cc34aa3a20c11c7302"}
Sep 17 18:10:23.383893 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:23.383726    2589 scope.go:117] "RemoveContainer" containerID="6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9"
Sep 17 18:10:23.392423 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:23.392405    2589 scope.go:117] "RemoveContainer" containerID="6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9"
Sep 17 18:10:23.392650 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:10:23.392632    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9\": container with ID starting with 6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9 not found: ID does not exist" containerID="6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9"
Sep 17 18:10:23.392695 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:23.392658    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9"} err="failed to get container status \"6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9\": rpc error: code = NotFound desc = could not find container \"6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9\": container with ID starting with 6931e8fe38da219f3b4fbcc85598ef70188e78948910f98e12e6c7e18b780ce9 not found: ID does not exist"
Sep 17 18:10:23.404356 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:23.404330    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9445698fd-8gqhf"]
Sep 17 18:10:23.411147 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:23.410222    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9445698fd-8gqhf"]
Sep 17 18:10:24.682227 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:24.682190    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="26f837ae-03db-494a-90d3-1afc2991651d" path="/var/lib/kubelet/pods/26f837ae-03db-494a-90d3-1afc2991651d/volumes"
Sep 17 18:10:27.409510 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:27.409482    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:27.510607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:27.510570    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:29.954457 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:29.954423    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:30.059441 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:30.059410    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:30.317028 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:30.316970    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:30.416182 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:30.416131    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:36.780284 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:36.780243    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:36.886778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:36.886735    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:41.598442 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:41.598415    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:41.692131 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:41.692102    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:44.062159 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:44.062131    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:44.163142 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:44.163099    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:45.986524 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:45.986494    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:10:46.464104 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:10:46.464069    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:00.503073 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.503029    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:00.595907 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.595867    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:00.680426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.680393    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:00.885130 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.885048    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:11:00.885323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.885312    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="26f837ae-03db-494a-90d3-1afc2991651d" containerName="console"
Sep 17 18:11:00.885369 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.885325    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="26f837ae-03db-494a-90d3-1afc2991651d" containerName="console"
Sep 17 18:11:00.885403 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.885379    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="26f837ae-03db-494a-90d3-1afc2991651d" containerName="console"
Sep 17 18:11:00.887049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.887033    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:00.889365 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.889346    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 17 18:11:00.898347 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.898328    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:11:00.937633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:00.937602    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:11:01.024976 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.024937    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4zr6\" (UniqueName: \"kubernetes.io/projected/be775eda-0f2f-4b43-be7d-0dcf19fe0bb4-kube-api-access-b4zr6\") pod \"limitador-limitador-84994f7ffc-v7sbj\" (UID: \"be775eda-0f2f-4b43-be7d-0dcf19fe0bb4\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:01.024976 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.024977    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/be775eda-0f2f-4b43-be7d-0dcf19fe0bb4-config-file\") pod \"limitador-limitador-84994f7ffc-v7sbj\" (UID: \"be775eda-0f2f-4b43-be7d-0dcf19fe0bb4\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:01.125760 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.125716    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4zr6\" (UniqueName: \"kubernetes.io/projected/be775eda-0f2f-4b43-be7d-0dcf19fe0bb4-kube-api-access-b4zr6\") pod \"limitador-limitador-84994f7ffc-v7sbj\" (UID: \"be775eda-0f2f-4b43-be7d-0dcf19fe0bb4\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:01.125760 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.125762    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/be775eda-0f2f-4b43-be7d-0dcf19fe0bb4-config-file\") pod \"limitador-limitador-84994f7ffc-v7sbj\" (UID: \"be775eda-0f2f-4b43-be7d-0dcf19fe0bb4\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:01.126367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.126351    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/be775eda-0f2f-4b43-be7d-0dcf19fe0bb4-config-file\") pod \"limitador-limitador-84994f7ffc-v7sbj\" (UID: \"be775eda-0f2f-4b43-be7d-0dcf19fe0bb4\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:01.133615 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.133597    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4zr6\" (UniqueName: \"kubernetes.io/projected/be775eda-0f2f-4b43-be7d-0dcf19fe0bb4-kube-api-access-b4zr6\") pod \"limitador-limitador-84994f7ffc-v7sbj\" (UID: \"be775eda-0f2f-4b43-be7d-0dcf19fe0bb4\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:01.196637 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.196577    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:01.324707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.324671    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5449"]
Sep 17 18:11:01.326845 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.326828    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449"
Sep 17 18:11:01.329271 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.329249    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-hsnsc\""
Sep 17 18:11:01.333282 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.333240    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5449"]
Sep 17 18:11:01.364853 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.364812    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:01.427463 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.427428    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvvcs\" (UniqueName: \"kubernetes.io/projected/59b80605-138e-4335-9abe-d91ad423a7c5-kube-api-access-zvvcs\") pod \"authorino-7bcfd8f9cd-n5449\" (UID: \"59b80605-138e-4335-9abe-d91ad423a7c5\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n5449"
Sep 17 18:11:01.462094 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.461970    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:01.469732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.469711    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:11:01.472502 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:11:01.472480    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe775eda_0f2f_4b43_be7d_0dcf19fe0bb4.slice/crio-8caad8f63492248e878422d65ae9d1fe05442d10cac9ab45d3a376f9ad80da36 WatchSource:0}: Error finding container 8caad8f63492248e878422d65ae9d1fe05442d10cac9ab45d3a376f9ad80da36: Status 404 returned error can't find the container with id 8caad8f63492248e878422d65ae9d1fe05442d10cac9ab45d3a376f9ad80da36
Sep 17 18:11:01.523228 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.523201    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-rc94j"]
Sep 17 18:11:01.525491 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.525475    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rc94j"
Sep 17 18:11:01.528415 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.528394    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvvcs\" (UniqueName: \"kubernetes.io/projected/59b80605-138e-4335-9abe-d91ad423a7c5-kube-api-access-zvvcs\") pod \"authorino-7bcfd8f9cd-n5449\" (UID: \"59b80605-138e-4335-9abe-d91ad423a7c5\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n5449"
Sep 17 18:11:01.535320 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.535300    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj" event={"ID":"be775eda-0f2f-4b43-be7d-0dcf19fe0bb4","Type":"ContainerStarted","Data":"8caad8f63492248e878422d65ae9d1fe05442d10cac9ab45d3a376f9ad80da36"}
Sep 17 18:11:01.538754 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.538736    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvvcs\" (UniqueName: \"kubernetes.io/projected/59b80605-138e-4335-9abe-d91ad423a7c5-kube-api-access-zvvcs\") pod \"authorino-7bcfd8f9cd-n5449\" (UID: \"59b80605-138e-4335-9abe-d91ad423a7c5\") " pod="kuadrant-system/authorino-7bcfd8f9cd-n5449"
Sep 17 18:11:01.544668 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.544641    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rc94j"]
Sep 17 18:11:01.628957 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.628927    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbxj\" (UniqueName: \"kubernetes.io/projected/6816ef1c-85b0-46e4-abb6-d1072227799c-kube-api-access-8hbxj\") pod \"authorino-59c974bd48-rc94j\" (UID: \"6816ef1c-85b0-46e4-abb6-d1072227799c\") " pod="kuadrant-system/authorino-59c974bd48-rc94j"
Sep 17 18:11:01.636854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.636826    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449"
Sep 17 18:11:01.730346 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.730262    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbxj\" (UniqueName: \"kubernetes.io/projected/6816ef1c-85b0-46e4-abb6-d1072227799c-kube-api-access-8hbxj\") pod \"authorino-59c974bd48-rc94j\" (UID: \"6816ef1c-85b0-46e4-abb6-d1072227799c\") " pod="kuadrant-system/authorino-59c974bd48-rc94j"
Sep 17 18:11:01.739169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.739143    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbxj\" (UniqueName: \"kubernetes.io/projected/6816ef1c-85b0-46e4-abb6-d1072227799c-kube-api-access-8hbxj\") pod \"authorino-59c974bd48-rc94j\" (UID: \"6816ef1c-85b0-46e4-abb6-d1072227799c\") " pod="kuadrant-system/authorino-59c974bd48-rc94j"
Sep 17 18:11:01.835252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.835204    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rc94j"
Sep 17 18:11:01.906878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:01.906845    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5449"]
Sep 17 18:11:01.910027 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:11:01.909982    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59b80605_138e_4335_9abe_d91ad423a7c5.slice/crio-e402d1afa374dc87220277f85985667f937df74302ac8756f2fe99d8c83bc286 WatchSource:0}: Error finding container e402d1afa374dc87220277f85985667f937df74302ac8756f2fe99d8c83bc286: Status 404 returned error can't find the container with id e402d1afa374dc87220277f85985667f937df74302ac8756f2fe99d8c83bc286
Sep 17 18:11:02.104057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:02.103955    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rc94j"]
Sep 17 18:11:02.106541 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:11:02.106519    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6816ef1c_85b0_46e4_abb6_d1072227799c.slice/crio-db9d1b327be471caf6d987a4a1b21a23547fbf65971ee55028ac8ef1ae86b6e2 WatchSource:0}: Error finding container db9d1b327be471caf6d987a4a1b21a23547fbf65971ee55028ac8ef1ae86b6e2: Status 404 returned error can't find the container with id db9d1b327be471caf6d987a4a1b21a23547fbf65971ee55028ac8ef1ae86b6e2
Sep 17 18:11:02.541757 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:02.541708    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rc94j" event={"ID":"6816ef1c-85b0-46e4-abb6-d1072227799c","Type":"ContainerStarted","Data":"db9d1b327be471caf6d987a4a1b21a23547fbf65971ee55028ac8ef1ae86b6e2"}
Sep 17 18:11:02.543261 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:02.543225    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449" event={"ID":"59b80605-138e-4335-9abe-d91ad423a7c5","Type":"ContainerStarted","Data":"e402d1afa374dc87220277f85985667f937df74302ac8756f2fe99d8c83bc286"}
Sep 17 18:11:05.561824 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:05.561787    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj" event={"ID":"be775eda-0f2f-4b43-be7d-0dcf19fe0bb4","Type":"ContainerStarted","Data":"256069f69fe045f4d3272f4111d83b25340e7beaa3619a5549af52fa2221659b"}
Sep 17 18:11:05.562246 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:05.561847    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:05.563414 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:05.563389    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rc94j" event={"ID":"6816ef1c-85b0-46e4-abb6-d1072227799c","Type":"ContainerStarted","Data":"62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396"}
Sep 17 18:11:05.564837 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:05.564816    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449" event={"ID":"59b80605-138e-4335-9abe-d91ad423a7c5","Type":"ContainerStarted","Data":"3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb"}
Sep 17 18:11:05.583214 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:05.583173    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj" podStartSLOduration=2.3711709499999998 podStartE2EDuration="5.583159671s" totalImagesPullingTime="3.211988721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:05.581505532 +0000 UTC m=+501.261191386" watchObservedRunningTime="2026-09-17 18:11:05.583159671 +0000 UTC m=+501.262845538"
Sep 17 18:11:05.598117 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:05.598078    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449" podStartSLOduration=1.821155095 podStartE2EDuration="4.598063776s" totalImagesPullingTime="2.776908681s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:05.597307514 +0000 UTC m=+501.276993369" watchObservedRunningTime="2026-09-17 18:11:05.598063776 +0000 UTC m=+501.277749630"
Sep 17 18:11:05.616492 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:05.616449    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-rc94j" podStartSLOduration=2.028690003 podStartE2EDuration="4.616436653s" totalImagesPullingTime="2.58774665s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:05.61499392 +0000 UTC m=+501.294679774" watchObservedRunningTime="2026-09-17 18:11:05.616436653 +0000 UTC m=+501.296122508"
Sep 17 18:11:05.652809 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:05.652781    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5449"]
Sep 17 18:11:07.572545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:07.572506    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449" podUID="59b80605-138e-4335-9abe-d91ad423a7c5" containerName="authorino" containerID="cri-o://3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb" gracePeriod=30
Sep 17 18:11:07.801958 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:07.801935    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449"
Sep 17 18:11:07.877950 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:07.877873    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zvvcs\" (UniqueName: \"kubernetes.io/projected/59b80605-138e-4335-9abe-d91ad423a7c5-kube-api-access-zvvcs\") pod \"59b80605-138e-4335-9abe-d91ad423a7c5\" (UID: \"59b80605-138e-4335-9abe-d91ad423a7c5\") "
Sep 17 18:11:07.880119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:07.880097    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59b80605-138e-4335-9abe-d91ad423a7c5-kube-api-access-zvvcs" (OuterVolumeSpecName: "kube-api-access-zvvcs") pod "59b80605-138e-4335-9abe-d91ad423a7c5" (UID: "59b80605-138e-4335-9abe-d91ad423a7c5"). InnerVolumeSpecName "kube-api-access-zvvcs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:07.979052 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:07.978986    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvvcs\" (UniqueName: \"kubernetes.io/projected/59b80605-138e-4335-9abe-d91ad423a7c5-kube-api-access-zvvcs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:11:08.577849 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.577815    2589 generic.go:356] "Generic (PLEG): container finished" podID="59b80605-138e-4335-9abe-d91ad423a7c5" containerID="3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb" exitCode=0
Sep 17 18:11:08.578266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.577859    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449"
Sep 17 18:11:08.578266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.577895    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449" event={"ID":"59b80605-138e-4335-9abe-d91ad423a7c5","Type":"ContainerDied","Data":"3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb"}
Sep 17 18:11:08.578266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.577927    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-n5449" event={"ID":"59b80605-138e-4335-9abe-d91ad423a7c5","Type":"ContainerDied","Data":"e402d1afa374dc87220277f85985667f937df74302ac8756f2fe99d8c83bc286"}
Sep 17 18:11:08.578266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.577945    2589 scope.go:117] "RemoveContainer" containerID="3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb"
Sep 17 18:11:08.586529 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.586511    2589 scope.go:117] "RemoveContainer" containerID="3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb"
Sep 17 18:11:08.586804 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:11:08.586788    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb\": container with ID starting with 3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb not found: ID does not exist" containerID="3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb"
Sep 17 18:11:08.586858 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.586811    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb"} err="failed to get container status \"3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb\": rpc error: code = NotFound desc = could not find container \"3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb\": container with ID starting with 3ef16aaf32edc6c3c5897d79b5aaa629b601b6260b263d5786cee12bff868dcb not found: ID does not exist"
Sep 17 18:11:08.616503 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.616478    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5449"]
Sep 17 18:11:08.623959 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.623941    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-n5449"]
Sep 17 18:11:08.682645 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:08.682607    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59b80605-138e-4335-9abe-d91ad423a7c5" path="/var/lib/kubelet/pods/59b80605-138e-4335-9abe-d91ad423a7c5/volumes"
Sep 17 18:11:16.570244 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:16.570214    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"
Sep 17 18:11:22.092320 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:22.092277    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:22.183298 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:22.183267    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:22.582824 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:22.582783    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:35.208266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.208167    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:35.364075 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.364032    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:11:35.419789 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.419757    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-79595"]
Sep 17 18:11:35.420075 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.420063    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59b80605-138e-4335-9abe-d91ad423a7c5" containerName="authorino"
Sep 17 18:11:35.420124 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.420077    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="59b80605-138e-4335-9abe-d91ad423a7c5" containerName="authorino"
Sep 17 18:11:35.420158 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.420135    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="59b80605-138e-4335-9abe-d91ad423a7c5" containerName="authorino"
Sep 17 18:11:35.422021 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.421987    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-79595"
Sep 17 18:11:35.430695 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.430665    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-79595"]
Sep 17 18:11:35.491097 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.491038    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-79595"]
Sep 17 18:11:35.491244 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:11:35.491227    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-nd8fr], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-79595" podUID="701a8376-38aa-4399-b1e0-797e161f954d"
Sep 17 18:11:35.494967 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.494948    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd8fr\" (UniqueName: \"kubernetes.io/projected/701a8376-38aa-4399-b1e0-797e161f954d-kube-api-access-nd8fr\") pod \"authorino-557f876f69-79595\" (UID: \"701a8376-38aa-4399-b1e0-797e161f954d\") " pod="kuadrant-system/authorino-557f876f69-79595"
Sep 17 18:11:35.522502 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.522478    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mn9ph"]
Sep 17 18:11:35.524917 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.524899    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:35.527309 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.527289    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 17 18:11:35.534319 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.534299    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mn9ph"]
Sep 17 18:11:35.595877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.595846    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10d8a41d-0188-4951-8d33-d10e26c46fd6-tls-cert\") pod \"authorino-6d4dfc649f-mn9ph\" (UID: \"10d8a41d-0188-4951-8d33-d10e26c46fd6\") " pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:35.595877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.595891    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd8fr\" (UniqueName: \"kubernetes.io/projected/701a8376-38aa-4399-b1e0-797e161f954d-kube-api-access-nd8fr\") pod \"authorino-557f876f69-79595\" (UID: \"701a8376-38aa-4399-b1e0-797e161f954d\") " pod="kuadrant-system/authorino-557f876f69-79595"
Sep 17 18:11:35.596155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.595912    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qwmz\" (UniqueName: \"kubernetes.io/projected/10d8a41d-0188-4951-8d33-d10e26c46fd6-kube-api-access-4qwmz\") pod \"authorino-6d4dfc649f-mn9ph\" (UID: \"10d8a41d-0188-4951-8d33-d10e26c46fd6\") " pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:35.603993 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.603971    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd8fr\" (UniqueName: \"kubernetes.io/projected/701a8376-38aa-4399-b1e0-797e161f954d-kube-api-access-nd8fr\") pod \"authorino-557f876f69-79595\" (UID: \"701a8376-38aa-4399-b1e0-797e161f954d\") " pod="kuadrant-system/authorino-557f876f69-79595"
Sep 17 18:11:35.653264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.653235    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mn9ph"]
Sep 17 18:11:35.653449 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:11:35.653433    2589 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4qwmz tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-mn9ph" podUID="10d8a41d-0188-4951-8d33-d10e26c46fd6"
Sep 17 18:11:35.683327 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.683300    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-78b46b448b-qp29g"]
Sep 17 18:11:35.685504 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.685484    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:35.685591 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.685484    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-79595"
Sep 17 18:11:35.685977 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.685961    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:11:35.691225 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.691207    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:35.695220 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.695198    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-79595"
Sep 17 18:11:35.695724 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.695705    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78b46b448b-qp29g"]
Sep 17 18:11:35.696251 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.696234    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10d8a41d-0188-4951-8d33-d10e26c46fd6-tls-cert\") pod \"authorino-6d4dfc649f-mn9ph\" (UID: \"10d8a41d-0188-4951-8d33-d10e26c46fd6\") " pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:35.696324 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.696277    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qwmz\" (UniqueName: \"kubernetes.io/projected/10d8a41d-0188-4951-8d33-d10e26c46fd6-kube-api-access-4qwmz\") pod \"authorino-6d4dfc649f-mn9ph\" (UID: \"10d8a41d-0188-4951-8d33-d10e26c46fd6\") " pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:35.698996 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.698968    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10d8a41d-0188-4951-8d33-d10e26c46fd6-tls-cert\") pod \"authorino-6d4dfc649f-mn9ph\" (UID: \"10d8a41d-0188-4951-8d33-d10e26c46fd6\") " pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:35.706580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.706562    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qwmz\" (UniqueName: \"kubernetes.io/projected/10d8a41d-0188-4951-8d33-d10e26c46fd6-kube-api-access-4qwmz\") pod \"authorino-6d4dfc649f-mn9ph\" (UID: \"10d8a41d-0188-4951-8d33-d10e26c46fd6\") " pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:35.797101 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.797074    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nd8fr\" (UniqueName: \"kubernetes.io/projected/701a8376-38aa-4399-b1e0-797e161f954d-kube-api-access-nd8fr\") pod \"701a8376-38aa-4399-b1e0-797e161f954d\" (UID: \"701a8376-38aa-4399-b1e0-797e161f954d\") "
Sep 17 18:11:35.797266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.797131    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10d8a41d-0188-4951-8d33-d10e26c46fd6-tls-cert\") pod \"10d8a41d-0188-4951-8d33-d10e26c46fd6\" (UID: \"10d8a41d-0188-4951-8d33-d10e26c46fd6\") "
Sep 17 18:11:35.797266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.797165    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qwmz\" (UniqueName: \"kubernetes.io/projected/10d8a41d-0188-4951-8d33-d10e26c46fd6-kube-api-access-4qwmz\") pod \"10d8a41d-0188-4951-8d33-d10e26c46fd6\" (UID: \"10d8a41d-0188-4951-8d33-d10e26c46fd6\") "
Sep 17 18:11:35.797378 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.797266    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0667284a-5615-4228-b26a-a39fd974644a-tls-cert\") pod \"authorino-78b46b448b-qp29g\" (UID: \"0667284a-5615-4228-b26a-a39fd974644a\") " pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:11:35.797378 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.797295    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9xv5\" (UniqueName: \"kubernetes.io/projected/0667284a-5615-4228-b26a-a39fd974644a-kube-api-access-v9xv5\") pod \"authorino-78b46b448b-qp29g\" (UID: \"0667284a-5615-4228-b26a-a39fd974644a\") " pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:11:35.799419 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.799388    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10d8a41d-0188-4951-8d33-d10e26c46fd6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "10d8a41d-0188-4951-8d33-d10e26c46fd6" (UID: "10d8a41d-0188-4951-8d33-d10e26c46fd6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:11:35.799547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.799422    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10d8a41d-0188-4951-8d33-d10e26c46fd6-kube-api-access-4qwmz" (OuterVolumeSpecName: "kube-api-access-4qwmz") pod "10d8a41d-0188-4951-8d33-d10e26c46fd6" (UID: "10d8a41d-0188-4951-8d33-d10e26c46fd6"). InnerVolumeSpecName "kube-api-access-4qwmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:35.799547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.799428    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/701a8376-38aa-4399-b1e0-797e161f954d-kube-api-access-nd8fr" (OuterVolumeSpecName: "kube-api-access-nd8fr") pod "701a8376-38aa-4399-b1e0-797e161f954d" (UID: "701a8376-38aa-4399-b1e0-797e161f954d"). InnerVolumeSpecName "kube-api-access-nd8fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:35.897811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.897783    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0667284a-5615-4228-b26a-a39fd974644a-tls-cert\") pod \"authorino-78b46b448b-qp29g\" (UID: \"0667284a-5615-4228-b26a-a39fd974644a\") " pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:11:35.897811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.897813    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9xv5\" (UniqueName: \"kubernetes.io/projected/0667284a-5615-4228-b26a-a39fd974644a-kube-api-access-v9xv5\") pod \"authorino-78b46b448b-qp29g\" (UID: \"0667284a-5615-4228-b26a-a39fd974644a\") " pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:11:35.897970 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.897859    2589 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/10d8a41d-0188-4951-8d33-d10e26c46fd6-tls-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.897970 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.897868    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qwmz\" (UniqueName: \"kubernetes.io/projected/10d8a41d-0188-4951-8d33-d10e26c46fd6-kube-api-access-4qwmz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.897970 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.897879    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd8fr\" (UniqueName: \"kubernetes.io/projected/701a8376-38aa-4399-b1e0-797e161f954d-kube-api-access-nd8fr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:11:35.900178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.900160    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0667284a-5615-4228-b26a-a39fd974644a-tls-cert\") pod \"authorino-78b46b448b-qp29g\" (UID: \"0667284a-5615-4228-b26a-a39fd974644a\") " pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:11:35.906576 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:35.906559    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9xv5\" (UniqueName: \"kubernetes.io/projected/0667284a-5615-4228-b26a-a39fd974644a-kube-api-access-v9xv5\") pod \"authorino-78b46b448b-qp29g\" (UID: \"0667284a-5615-4228-b26a-a39fd974644a\") " pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:11:36.008487 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:36.008455    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:11:36.283356 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:36.283324    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-78b46b448b-qp29g"]
Sep 17 18:11:36.286507 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:11:36.286481    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0667284a_5615_4228_b26a_a39fd974644a.slice/crio-ac93f921344994a4ba17a19125c5edc9f4e796e1e058078faeff2f8a40bf40b4 WatchSource:0}: Error finding container ac93f921344994a4ba17a19125c5edc9f4e796e1e058078faeff2f8a40bf40b4: Status 404 returned error can't find the container with id ac93f921344994a4ba17a19125c5edc9f4e796e1e058078faeff2f8a40bf40b4
Sep 17 18:11:36.690498 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:36.690466    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b46b448b-qp29g" event={"ID":"0667284a-5615-4228-b26a-a39fd974644a","Type":"ContainerStarted","Data":"ac93f921344994a4ba17a19125c5edc9f4e796e1e058078faeff2f8a40bf40b4"}
Sep 17 18:11:36.690498 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:36.690494    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-79595"
Sep 17 18:11:36.690667 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:36.690501    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-mn9ph"
Sep 17 18:11:36.721457 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:36.721431    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mn9ph"]
Sep 17 18:11:36.727261 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:36.727236    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-mn9ph"]
Sep 17 18:11:36.759284 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:36.759262    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-79595"]
Sep 17 18:11:36.768544 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:36.768520    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-79595"]
Sep 17 18:11:38.682954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:38.682921    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10d8a41d-0188-4951-8d33-d10e26c46fd6" path="/var/lib/kubelet/pods/10d8a41d-0188-4951-8d33-d10e26c46fd6/volumes"
Sep 17 18:11:38.683326 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:38.683150    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="701a8376-38aa-4399-b1e0-797e161f954d" path="/var/lib/kubelet/pods/701a8376-38aa-4399-b1e0-797e161f954d/volumes"
Sep 17 18:11:57.777050 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:57.776938    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b46b448b-qp29g" event={"ID":"0667284a-5615-4228-b26a-a39fd974644a","Type":"ContainerStarted","Data":"6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44"}
Sep 17 18:11:57.794874 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:57.794825    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-78b46b448b-qp29g" podStartSLOduration=1.5567072130000001 podStartE2EDuration="22.794809386s" totalImagesPullingTime="21.238102173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:11:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:11:57.793491868 +0000 UTC m=+553.473177722" watchObservedRunningTime="2026-09-17 18:11:57.794809386 +0000 UTC m=+553.474495240"
Sep 17 18:11:57.835250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:57.835217    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rc94j"]
Sep 17 18:11:57.835421 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:57.835399    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-rc94j" podUID="6816ef1c-85b0-46e4-abb6-d1072227799c" containerName="authorino" containerID="cri-o://62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396" gracePeriod=30
Sep 17 18:11:58.067836 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.067811    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rc94j"
Sep 17 18:11:58.162202 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.162163    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hbxj\" (UniqueName: \"kubernetes.io/projected/6816ef1c-85b0-46e4-abb6-d1072227799c-kube-api-access-8hbxj\") pod \"6816ef1c-85b0-46e4-abb6-d1072227799c\" (UID: \"6816ef1c-85b0-46e4-abb6-d1072227799c\") "
Sep 17 18:11:58.164385 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.164362    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6816ef1c-85b0-46e4-abb6-d1072227799c-kube-api-access-8hbxj" (OuterVolumeSpecName: "kube-api-access-8hbxj") pod "6816ef1c-85b0-46e4-abb6-d1072227799c" (UID: "6816ef1c-85b0-46e4-abb6-d1072227799c"). InnerVolumeSpecName "kube-api-access-8hbxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:11:58.263045 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.262990    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbxj\" (UniqueName: \"kubernetes.io/projected/6816ef1c-85b0-46e4-abb6-d1072227799c-kube-api-access-8hbxj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:11:58.782982 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.782948    2589 generic.go:356] "Generic (PLEG): container finished" podID="6816ef1c-85b0-46e4-abb6-d1072227799c" containerID="62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396" exitCode=0
Sep 17 18:11:58.783495 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.782997    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-rc94j"
Sep 17 18:11:58.783495 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.783032    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rc94j" event={"ID":"6816ef1c-85b0-46e4-abb6-d1072227799c","Type":"ContainerDied","Data":"62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396"}
Sep 17 18:11:58.783495 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.783074    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-rc94j" event={"ID":"6816ef1c-85b0-46e4-abb6-d1072227799c","Type":"ContainerDied","Data":"db9d1b327be471caf6d987a4a1b21a23547fbf65971ee55028ac8ef1ae86b6e2"}
Sep 17 18:11:58.783495 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.783091    2589 scope.go:117] "RemoveContainer" containerID="62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396"
Sep 17 18:11:58.791181 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.791112    2589 scope.go:117] "RemoveContainer" containerID="62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396"
Sep 17 18:11:58.791406 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:11:58.791389    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396\": container with ID starting with 62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396 not found: ID does not exist" containerID="62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396"
Sep 17 18:11:58.791454 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.791415    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396"} err="failed to get container status \"62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396\": rpc error: code = NotFound desc = could not find container \"62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396\": container with ID starting with 62ac52cb80f123512534620b56ba54060f1be680705e714f16318535ff21f396 not found: ID does not exist"
Sep 17 18:11:58.811296 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.811240    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rc94j"]
Sep 17 18:11:58.812585 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:11:58.812566    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-rc94j"]
Sep 17 18:12:00.682639 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:00.682601    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6816ef1c-85b0-46e4-abb6-d1072227799c" path="/var/lib/kubelet/pods/6816ef1c-85b0-46e4-abb6-d1072227799c/volumes"
Sep 17 18:12:02.707621 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:02.707593    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:02.803657 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:02.803626    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:02.934926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:02.934892    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:03.113812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:03.113779    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:03.200686 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:03.200651    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:06.432212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:06.432183    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:06.531803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:06.531766    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:35.348936 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:35.348904    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:39.110602 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:39.110568    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:39.270477 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:39.270446    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:40.504029 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:40.503969    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:40.603471 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:40.603434    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:40.703664 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:40.703614    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:40.803515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:40.803426    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:41.036958 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:41.036927    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:41.140593 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:41.140490    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:41.228906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:41.228874    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:12:46.568273 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:46.568243    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:12:46.571204 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:46.571181    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:12:51.427548 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:51.427518    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:51.550263 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:51.550209    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:12:59.328206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:12:59.328108    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:10.038027 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:10.037951    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:10.540783 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:10.540745    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:13:14.273275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:14.273235    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:14.347511 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:14.347480    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:20.171569 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:20.171532    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:20.941615 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:20.941579    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:13:21.210805 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:21.210728    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:32.398622 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:32.398559    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:33.346114 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:33.346077    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:13:40.731258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:40.731220    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:40.834198 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:40.834157    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:41.911435 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.911406    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"]
Sep 17 18:13:41.911861 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.911691    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6816ef1c-85b0-46e4-abb6-d1072227799c" containerName="authorino"
Sep 17 18:13:41.911861 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.911703    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="6816ef1c-85b0-46e4-abb6-d1072227799c" containerName="authorino"
Sep 17 18:13:41.911861 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.911765    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="6816ef1c-85b0-46e4-abb6-d1072227799c" containerName="authorino"
Sep 17 18:13:41.914554 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.914538    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:41.917942 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.917921    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 17 18:13:41.918948 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.918929    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 17 18:13:41.919048 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.918999    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k9rwt\""
Sep 17 18:13:41.919048 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.918999    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 17 18:13:41.926968 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.926950    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"]
Sep 17 18:13:41.945577 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:41.945553    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:42.036529 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.036495    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.036690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.036536    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6z7s\" (UniqueName: \"kubernetes.io/projected/96f9bf13-92f4-42e9-b91f-48702dd6a833-kube-api-access-w6z7s\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.036690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.036566    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f9bf13-92f4-42e9-b91f-48702dd6a833-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.036690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.036633    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.036690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.036660    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.036690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.036684    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.137375 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.137341    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6z7s\" (UniqueName: \"kubernetes.io/projected/96f9bf13-92f4-42e9-b91f-48702dd6a833-kube-api-access-w6z7s\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.137509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.137381    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f9bf13-92f4-42e9-b91f-48702dd6a833-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.137568 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.137506    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.137568 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.137538    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.137568 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.137556    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.137715 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.137604    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.137928 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.137901    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.138026 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.137899    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.138026 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.137969    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.139803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.139781    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96f9bf13-92f4-42e9-b91f-48702dd6a833-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.140211 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.140192    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96f9bf13-92f4-42e9-b91f-48702dd6a833-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.149942 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.149922    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6z7s\" (UniqueName: \"kubernetes.io/projected/96f9bf13-92f4-42e9-b91f-48702dd6a833-kube-api-access-w6z7s\") pod \"e2e-trlp-test-simulated-kserve-598466548d-ggnnw\" (UID: \"96f9bf13-92f4-42e9-b91f-48702dd6a833\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.224780 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.224704    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:42.496847 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.496765    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"]
Sep 17 18:13:42.500592 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:13:42.500564    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f9bf13_92f4_42e9_b91f_48702dd6a833.slice/crio-9f958d7f4f1149fad078e460a16363262b707fd30ba4b46f7796d53f4f437c4e WatchSource:0}: Error finding container 9f958d7f4f1149fad078e460a16363262b707fd30ba4b46f7796d53f4f437c4e: Status 404 returned error can't find the container with id 9f958d7f4f1149fad078e460a16363262b707fd30ba4b46f7796d53f4f437c4e
Sep 17 18:13:42.501878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:42.501864    2589 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:13:43.204397 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:43.204357    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw" event={"ID":"96f9bf13-92f4-42e9-b91f-48702dd6a833","Type":"ContainerStarted","Data":"9f958d7f4f1149fad078e460a16363262b707fd30ba4b46f7796d53f4f437c4e"}
Sep 17 18:13:43.308205 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:43.308162    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:13:44.210122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:44.210040    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw" event={"ID":"96f9bf13-92f4-42e9-b91f-48702dd6a833","Type":"ContainerStarted","Data":"76105223f5b0f662c3c76ad537e4e7aaab77180b43d83222668e3d9976594392"}
Sep 17 18:13:44.231677 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:44.231627    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw" podStartSLOduration=1.917092544 podStartE2EDuration="3.231611475s" totalImagesPullingTime="1.314518931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:44.230604899 +0000 UTC m=+659.910290754" watchObservedRunningTime="2026-09-17 18:13:44.231611475 +0000 UTC m=+659.911297328"
Sep 17 18:13:50.821536 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:50.821492    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:52.225398 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:52.225357    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:52.225766 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:52.225443    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:52.237760 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:52.237735    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:52.256130 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:52.256103    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-ggnnw"
Sep 17 18:13:54.017870 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.017826    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"]
Sep 17 18:13:54.023925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.023908    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.026645 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.026627    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 17 18:13:54.035474 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.035453    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"]
Sep 17 18:13:54.059159 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.059120    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:54.130750 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.130724    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.130854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.130771    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.130854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.130793    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a437c422-6d2a-4493-9265-dab5e2e510ba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.130928 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.130887    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.130928 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.130916    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.130992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.130942    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf9f4\" (UniqueName: \"kubernetes.io/projected/a437c422-6d2a-4493-9265-dab5e2e510ba-kube-api-access-kf9f4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.231477 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.231448    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.231620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.231485    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf9f4\" (UniqueName: \"kubernetes.io/projected/a437c422-6d2a-4493-9265-dab5e2e510ba-kube-api-access-kf9f4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.231620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.231518    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.231620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.231555    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.231620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.231573    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a437c422-6d2a-4493-9265-dab5e2e510ba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.231620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.231603    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.231932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.231895    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.231932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.231920    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.232106 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.231963    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.233920 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.233898    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a437c422-6d2a-4493-9265-dab5e2e510ba-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.234410 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.234387    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a437c422-6d2a-4493-9265-dab5e2e510ba-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.240558 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.240535    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf9f4\" (UniqueName: \"kubernetes.io/projected/a437c422-6d2a-4493-9265-dab5e2e510ba-kube-api-access-kf9f4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5\" (UID: \"a437c422-6d2a-4493-9265-dab5e2e510ba\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.334423 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.334337    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:13:54.606192 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:54.605979    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"]
Sep 17 18:13:54.608808 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:13:54.608780    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda437c422_6d2a_4493_9265_dab5e2e510ba.slice/crio-96b210de9856d074d21ca5984f2332bd12d0597cf2e85c63f3311ab157be96f2 WatchSource:0}: Error finding container 96b210de9856d074d21ca5984f2332bd12d0597cf2e85c63f3311ab157be96f2: Status 404 returned error can't find the container with id 96b210de9856d074d21ca5984f2332bd12d0597cf2e85c63f3311ab157be96f2
Sep 17 18:13:55.258357 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:55.258322    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5" event={"ID":"a437c422-6d2a-4493-9265-dab5e2e510ba","Type":"ContainerStarted","Data":"b93c4e47b0167b7149fd3d8e06e4e12813ecd593d83cc1a388bbca7acad7ef08"}
Sep 17 18:13:55.258822 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:55.258381    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5" event={"ID":"a437c422-6d2a-4493-9265-dab5e2e510ba","Type":"ContainerStarted","Data":"96b210de9856d074d21ca5984f2332bd12d0597cf2e85c63f3311ab157be96f2"}
Sep 17 18:13:55.278298 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:55.278255    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5" podStartSLOduration=2.059578219 podStartE2EDuration="2.278240243s" totalImagesPullingTime="218.662024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:13:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:13:55.278101084 +0000 UTC m=+670.957786938" watchObservedRunningTime="2026-09-17 18:13:55.278240243 +0000 UTC m=+670.957926096"
Sep 17 18:13:55.654076 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:55.653976    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:13:57.425197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:57.425167    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:13:57.493557 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:13:57.493511    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:04.334967 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:04.334930    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:14:04.335383 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:04.334982    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:14:04.347604 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:04.347582    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:14:05.309209 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:05.309181    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5"
Sep 17 18:14:35.405628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.405538    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"]
Sep 17 18:14:35.409057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.409033    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.411543 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.411521    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Sep 17 18:14:35.418178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.418153    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"]
Sep 17 18:14:35.429487 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.429465    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:35.455648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.455610    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcsp\" (UniqueName: \"kubernetes.io/projected/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-kube-api-access-nqcsp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.455648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.455651    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.455814 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.455677    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.455814 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.455789    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.455922 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.455823    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.455922 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.455892    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.556661 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.556614    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.556817 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.556679    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcsp\" (UniqueName: \"kubernetes.io/projected/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-kube-api-access-nqcsp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.556817 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.556712    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.556817 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.556748    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.556817 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.556802    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.557068 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.556825    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.557131 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.557089    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.557131 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.557105    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.557250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.557227    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.559247 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.559228    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.559423 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.559406    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.567528 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.567502    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcsp\" (UniqueName: \"kubernetes.io/projected/a601e0a9-6c9f-4970-ace4-a20b1c8edf7b-kube-api-access-nqcsp\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-w7mt2\" (UID: \"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.719709 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.719646    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:35.990664 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:35.990642    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"]
Sep 17 18:14:35.993342 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:14:35.993317    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda601e0a9_6c9f_4970_ace4_a20b1c8edf7b.slice/crio-67912fc64e71e5e844e77688956a3993d5ec958f8d29eaa6ceeb4c81117a898c WatchSource:0}: Error finding container 67912fc64e71e5e844e77688956a3993d5ec958f8d29eaa6ceeb4c81117a898c: Status 404 returned error can't find the container with id 67912fc64e71e5e844e77688956a3993d5ec958f8d29eaa6ceeb4c81117a898c
Sep 17 18:14:36.433753 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:36.433717    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2" event={"ID":"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b","Type":"ContainerStarted","Data":"d4a1fdcff26d60f69364302a389f1afc316a83e5d7731c65c88473c1f21b1399"}
Sep 17 18:14:36.434221 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:36.433764    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2" event={"ID":"a601e0a9-6c9f-4970-ace4-a20b1c8edf7b","Type":"ContainerStarted","Data":"67912fc64e71e5e844e77688956a3993d5ec958f8d29eaa6ceeb4c81117a898c"}
Sep 17 18:14:36.454668 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:36.454610    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2" podStartSLOduration=1.271379764 podStartE2EDuration="1.454594417s" totalImagesPullingTime="183.214653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:36.454299676 +0000 UTC m=+712.133985531" watchObservedRunningTime="2026-09-17 18:14:36.454594417 +0000 UTC m=+712.134280270"
Sep 17 18:14:37.304187 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:37.304147    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:14:38.327587 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:38.327543    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:45.719765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:45.719728    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:45.720195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:45.720040    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:45.732455 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:45.732431    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:46.487953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.487924    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-w7mt2"
Sep 17 18:14:46.595100 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.595063    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"]
Sep 17 18:14:46.598616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.598597    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.600895 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.600875    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 17 18:14:46.611839 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.611815    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"]
Sep 17 18:14:46.618297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.618275    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:14:46.748744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.748657    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.748744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.748694    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.748744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.748715    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80d2928-2b56-4a0c-b295-ac9de7190d22-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.749273 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.748770    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.749273 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.748817    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.749273 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.748890    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt5c2\" (UniqueName: \"kubernetes.io/projected/b80d2928-2b56-4a0c-b295-ac9de7190d22-kube-api-access-dt5c2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.850229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.850198    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.850229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.850234    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.850436 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.850251    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80d2928-2b56-4a0c-b295-ac9de7190d22-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.850436 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.850270    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.850436 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.850315    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.850436 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.850344    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt5c2\" (UniqueName: \"kubernetes.io/projected/b80d2928-2b56-4a0c-b295-ac9de7190d22-kube-api-access-dt5c2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.850839 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.850811    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.851132 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.850827    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.851132 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.850857    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.852690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.852672    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b80d2928-2b56-4a0c-b295-ac9de7190d22-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.852771 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.852698    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b80d2928-2b56-4a0c-b295-ac9de7190d22-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.858882 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.858861    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt5c2\" (UniqueName: \"kubernetes.io/projected/b80d2928-2b56-4a0c-b295-ac9de7190d22-kube-api-access-dt5c2\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-n8795\" (UID: \"b80d2928-2b56-4a0c-b295-ac9de7190d22\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:46.908898 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:46.908876    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:47.181093 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:47.180855    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"]
Sep 17 18:14:47.183716 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:14:47.183689    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb80d2928_2b56_4a0c_b295_ac9de7190d22.slice/crio-4f217eb2dfaa5b827d9a131a81ea295ba62f573384bb890a1629f28a2957fad0 WatchSource:0}: Error finding container 4f217eb2dfaa5b827d9a131a81ea295ba62f573384bb890a1629f28a2957fad0: Status 404 returned error can't find the container with id 4f217eb2dfaa5b827d9a131a81ea295ba62f573384bb890a1629f28a2957fad0
Sep 17 18:14:47.483632 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:47.483556    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795" event={"ID":"b80d2928-2b56-4a0c-b295-ac9de7190d22","Type":"ContainerStarted","Data":"45e9cf1ed94dd266975400e3ee9eb67156bf99298bc44a1535999b1e4c255b31"}
Sep 17 18:14:47.483632 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:47.483605    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795" event={"ID":"b80d2928-2b56-4a0c-b295-ac9de7190d22","Type":"ContainerStarted","Data":"4f217eb2dfaa5b827d9a131a81ea295ba62f573384bb890a1629f28a2957fad0"}
Sep 17 18:14:47.500972 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:47.500921    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795" podStartSLOduration=1.325643202 podStartE2EDuration="1.500902536s" totalImagesPullingTime="175.259334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:14:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:14:47.500030696 +0000 UTC m=+723.179716552" watchObservedRunningTime="2026-09-17 18:14:47.500902536 +0000 UTC m=+723.180588390"
Sep 17 18:14:49.027377 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:49.027340    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:14:56.909714 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:56.909668    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:56.910206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:56.909737    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:56.922096 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:56.922074    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:14:57.538100 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:14:57.538072    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-n8795"
Sep 17 18:15:40.060987 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:40.060943    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:52.001664 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.001630    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-697d77b6c5-976hh"]
Sep 17 18:15:52.005016 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.004985    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-697d77b6c5-976hh"
Sep 17 18:15:52.013096 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.013074    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-697d77b6c5-976hh"]
Sep 17 18:15:52.091761 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.091729    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d91f7c4-3f84-4f6d-bc31-941201484f7c-tls-cert\") pod \"authorino-697d77b6c5-976hh\" (UID: \"5d91f7c4-3f84-4f6d-bc31-941201484f7c\") " pod="kuadrant-system/authorino-697d77b6c5-976hh"
Sep 17 18:15:52.091899 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.091786    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9dpl\" (UniqueName: \"kubernetes.io/projected/5d91f7c4-3f84-4f6d-bc31-941201484f7c-kube-api-access-x9dpl\") pod \"authorino-697d77b6c5-976hh\" (UID: \"5d91f7c4-3f84-4f6d-bc31-941201484f7c\") " pod="kuadrant-system/authorino-697d77b6c5-976hh"
Sep 17 18:15:52.192301 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.192268    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d91f7c4-3f84-4f6d-bc31-941201484f7c-tls-cert\") pod \"authorino-697d77b6c5-976hh\" (UID: \"5d91f7c4-3f84-4f6d-bc31-941201484f7c\") " pod="kuadrant-system/authorino-697d77b6c5-976hh"
Sep 17 18:15:52.192459 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.192311    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9dpl\" (UniqueName: \"kubernetes.io/projected/5d91f7c4-3f84-4f6d-bc31-941201484f7c-kube-api-access-x9dpl\") pod \"authorino-697d77b6c5-976hh\" (UID: \"5d91f7c4-3f84-4f6d-bc31-941201484f7c\") " pod="kuadrant-system/authorino-697d77b6c5-976hh"
Sep 17 18:15:52.194851 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.194823    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5d91f7c4-3f84-4f6d-bc31-941201484f7c-tls-cert\") pod \"authorino-697d77b6c5-976hh\" (UID: \"5d91f7c4-3f84-4f6d-bc31-941201484f7c\") " pod="kuadrant-system/authorino-697d77b6c5-976hh"
Sep 17 18:15:52.200820 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.200794    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9dpl\" (UniqueName: \"kubernetes.io/projected/5d91f7c4-3f84-4f6d-bc31-941201484f7c-kube-api-access-x9dpl\") pod \"authorino-697d77b6c5-976hh\" (UID: \"5d91f7c4-3f84-4f6d-bc31-941201484f7c\") " pod="kuadrant-system/authorino-697d77b6c5-976hh"
Sep 17 18:15:52.314749 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.314720    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-697d77b6c5-976hh"
Sep 17 18:15:52.586652 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.586485    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-697d77b6c5-976hh"]
Sep 17 18:15:52.589919 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:15:52.589887    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d91f7c4_3f84_4f6d_bc31_941201484f7c.slice/crio-90fc4c011e0be38a7e1d5d9ad7ba01e79e60363837b4c91fa7d0be1def9e5d85 WatchSource:0}: Error finding container 90fc4c011e0be38a7e1d5d9ad7ba01e79e60363837b4c91fa7d0be1def9e5d85: Status 404 returned error can't find the container with id 90fc4c011e0be38a7e1d5d9ad7ba01e79e60363837b4c91fa7d0be1def9e5d85
Sep 17 18:15:52.751871 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:52.751839    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-697d77b6c5-976hh" event={"ID":"5d91f7c4-3f84-4f6d-bc31-941201484f7c","Type":"ContainerStarted","Data":"90fc4c011e0be38a7e1d5d9ad7ba01e79e60363837b4c91fa7d0be1def9e5d85"}
Sep 17 18:15:54.763999 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:54.763953    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-697d77b6c5-976hh" event={"ID":"5d91f7c4-3f84-4f6d-bc31-941201484f7c","Type":"ContainerStarted","Data":"8c03e7fdd54dfa73317758d2bb549e203c8f437bc4298dcb3b75af9c11d73928"}
Sep 17 18:15:54.781277 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:54.781180    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-697d77b6c5-976hh" podStartSLOduration=2.500258967 podStartE2EDuration="3.781163764s" totalImagesPullingTime="1.280904797s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:15:54.780420588 +0000 UTC m=+790.460106442" watchObservedRunningTime="2026-09-17 18:15:54.781163764 +0000 UTC m=+790.460849670"
Sep 17 18:15:54.809678 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:54.809635    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78b46b448b-qp29g"]
Sep 17 18:15:54.809928 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:54.809899    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-78b46b448b-qp29g" podUID="0667284a-5615-4228-b26a-a39fd974644a" containerName="authorino" containerID="cri-o://6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44" gracePeriod=30
Sep 17 18:15:55.072358 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.072333    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:15:55.221517 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.221484    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0667284a-5615-4228-b26a-a39fd974644a-tls-cert\") pod \"0667284a-5615-4228-b26a-a39fd974644a\" (UID: \"0667284a-5615-4228-b26a-a39fd974644a\") "
Sep 17 18:15:55.221706 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.221566    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9xv5\" (UniqueName: \"kubernetes.io/projected/0667284a-5615-4228-b26a-a39fd974644a-kube-api-access-v9xv5\") pod \"0667284a-5615-4228-b26a-a39fd974644a\" (UID: \"0667284a-5615-4228-b26a-a39fd974644a\") "
Sep 17 18:15:55.223804 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.223763    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0667284a-5615-4228-b26a-a39fd974644a-kube-api-access-v9xv5" (OuterVolumeSpecName: "kube-api-access-v9xv5") pod "0667284a-5615-4228-b26a-a39fd974644a" (UID: "0667284a-5615-4228-b26a-a39fd974644a"). InnerVolumeSpecName "kube-api-access-v9xv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:15:55.231612 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.231584    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0667284a-5615-4228-b26a-a39fd974644a-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "0667284a-5615-4228-b26a-a39fd974644a" (UID: "0667284a-5615-4228-b26a-a39fd974644a"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:15:55.322722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.322638    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9xv5\" (UniqueName: \"kubernetes.io/projected/0667284a-5615-4228-b26a-a39fd974644a-kube-api-access-v9xv5\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:15:55.322722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.322679    2589 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/0667284a-5615-4228-b26a-a39fd974644a-tls-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:15:55.336333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.336284    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:55.437390 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.437354    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:55.769984 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.769946    2589 generic.go:356] "Generic (PLEG): container finished" podID="0667284a-5615-4228-b26a-a39fd974644a" containerID="6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44" exitCode=0
Sep 17 18:15:55.770417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.770024    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-78b46b448b-qp29g"
Sep 17 18:15:55.770417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.770049    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b46b448b-qp29g" event={"ID":"0667284a-5615-4228-b26a-a39fd974644a","Type":"ContainerDied","Data":"6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44"}
Sep 17 18:15:55.770417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.770094    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-78b46b448b-qp29g" event={"ID":"0667284a-5615-4228-b26a-a39fd974644a","Type":"ContainerDied","Data":"ac93f921344994a4ba17a19125c5edc9f4e796e1e058078faeff2f8a40bf40b4"}
Sep 17 18:15:55.770417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.770118    2589 scope.go:117] "RemoveContainer" containerID="6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44"
Sep 17 18:15:55.781091 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.781070    2589 scope.go:117] "RemoveContainer" containerID="6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44"
Sep 17 18:15:55.781395 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:15:55.781374    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44\": container with ID starting with 6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44 not found: ID does not exist" containerID="6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44"
Sep 17 18:15:55.781481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.781407    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44"} err="failed to get container status \"6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44\": rpc error: code = NotFound desc = could not find container \"6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44\": container with ID starting with 6ee8abf59e4ba763da7f156f4deaaba61a4e4bc567dcb42c4492e94d91ecad44 not found: ID does not exist"
Sep 17 18:15:55.793139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.793115    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-78b46b448b-qp29g"]
Sep 17 18:15:55.797548 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:55.797523    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-78b46b448b-qp29g"]
Sep 17 18:15:56.469323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:56.469286    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:56.561892 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:56.561848    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:15:56.684992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:15:56.684958    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0667284a-5615-4228-b26a-a39fd974644a" path="/var/lib/kubelet/pods/0667284a-5615-4228-b26a-a39fd974644a/volumes"
Sep 17 18:16:19.267282 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:19.267239    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:19.369383 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:19.369347    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:19.467706 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:19.467669    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:19.591074 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:19.590979    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:19.682764 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:19.682734    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:19.751743 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:19.751704    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:19.877955 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:19.877867    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:19.994931 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:19.994896    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:20.165478 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:20.165382    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:20.693781 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:20.693747    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:20.780423 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:20.780394    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:21.135074 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:21.135037    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:22.716671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:22.716639    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:22.829852 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:22.829815    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:22.936339 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:22.936294    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:23.041330 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:23.041285    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:23.133699 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:23.133663    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:23.407544 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:23.407450    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:16:24.040623 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:24.040593    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:24.143720 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:24.143670    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:24.243121 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:24.243083    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:24.699732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:24.699700    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:24.790254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:24.790223    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:25.227502 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.227469    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7640-961c34"]
Sep 17 18:16:25.227866 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.227771    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0667284a-5615-4228-b26a-a39fd974644a" containerName="authorino"
Sep 17 18:16:25.227866 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.227781    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="0667284a-5615-4228-b26a-a39fd974644a" containerName="authorino"
Sep 17 18:16:25.227866 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.227832    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="0667284a-5615-4228-b26a-a39fd974644a" containerName="authorino"
Sep 17 18:16:25.230845 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.230829    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7640-961c34"
Sep 17 18:16:25.233265 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.233244    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s5zgf\""
Sep 17 18:16:25.235343 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.235322    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7640-961c34"]
Sep 17 18:16:25.351500 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.351468    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:25.372467 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.372443    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjrfg\" (UniqueName: \"kubernetes.io/projected/2cf05ba8-4543-45c0-ac02-1120b14b19fe-kube-api-access-bjrfg\") pod \"test-curl-7640-961c34\" (UID: \"2cf05ba8-4543-45c0-ac02-1120b14b19fe\") " pod="opendatahub/test-curl-7640-961c34"
Sep 17 18:16:25.431548 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.431523    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:25.472812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.472786    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjrfg\" (UniqueName: \"kubernetes.io/projected/2cf05ba8-4543-45c0-ac02-1120b14b19fe-kube-api-access-bjrfg\") pod \"test-curl-7640-961c34\" (UID: \"2cf05ba8-4543-45c0-ac02-1120b14b19fe\") " pod="opendatahub/test-curl-7640-961c34"
Sep 17 18:16:25.482063 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.481964    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjrfg\" (UniqueName: \"kubernetes.io/projected/2cf05ba8-4543-45c0-ac02-1120b14b19fe-kube-api-access-bjrfg\") pod \"test-curl-7640-961c34\" (UID: \"2cf05ba8-4543-45c0-ac02-1120b14b19fe\") " pod="opendatahub/test-curl-7640-961c34"
Sep 17 18:16:25.502496 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.502467    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:25.540515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.540486    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7640-961c34"
Sep 17 18:16:25.808792 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.808717    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7640-961c34"]
Sep 17 18:16:25.811169 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:16:25.811135    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2cf05ba8_4543_45c0_ac02_1120b14b19fe.slice/crio-7fb9ecff2890d1f2c6a9496db5f5d800826e9cc45c12bcf8b54b202f8b3588c1 WatchSource:0}: Error finding container 7fb9ecff2890d1f2c6a9496db5f5d800826e9cc45c12bcf8b54b202f8b3588c1: Status 404 returned error can't find the container with id 7fb9ecff2890d1f2c6a9496db5f5d800826e9cc45c12bcf8b54b202f8b3588c1
Sep 17 18:16:25.893393 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.893362    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7640-961c34" event={"ID":"2cf05ba8-4543-45c0-ac02-1120b14b19fe","Type":"ContainerStarted","Data":"7fb9ecff2890d1f2c6a9496db5f5d800826e9cc45c12bcf8b54b202f8b3588c1"}
Sep 17 18:16:25.964322 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:25.964276    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:26.060290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:26.060211    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:26.447404 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:26.447322    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:26.547026 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:26.546965    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:26.692309 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:26.692274    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:27.075125 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:27.075078    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:27.168567 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:27.168528    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:27.539631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:27.539601    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:28.504071 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:28.504026    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:28.909542 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:28.909457    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7640-961c34" event={"ID":"2cf05ba8-4543-45c0-ac02-1120b14b19fe","Type":"ContainerStarted","Data":"a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f"}
Sep 17 18:16:28.925023 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:28.924964    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7640-961c34" podStartSLOduration=1.523413678 podStartE2EDuration="3.92495009s" totalImagesPullingTime="2.401536412s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:28.924102529 +0000 UTC m=+824.603788384" watchObservedRunningTime="2026-09-17 18:16:28.92495009 +0000 UTC m=+824.604635944"
Sep 17 18:16:29.391362 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:29.391321    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:29.490512 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:29.490482    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:29.515710 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:29.515678    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7640-961c34"]
Sep 17 18:16:29.518258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:29.518220    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7640-961c34"]
Sep 17 18:16:29.591295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:29.591216    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:29.900921 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:29.900837    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:29.913606 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:16:29.913580    2589 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7640-961c34" container="test-curl-7640-961c34"
Sep 17 18:16:29.998559 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:29.998517    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:30.098504 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.098470    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:30.282882 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.282842    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"]
Sep 17 18:16:30.283201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.283188    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2cf05ba8-4543-45c0-ac02-1120b14b19fe" containerName="test-curl-7640-961c34"
Sep 17 18:16:30.283250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.283203    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="2cf05ba8-4543-45c0-ac02-1120b14b19fe" containerName="test-curl-7640-961c34"
Sep 17 18:16:30.283285 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.283277    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="2cf05ba8-4543-45c0-ac02-1120b14b19fe" containerName="test-curl-7640-961c34"
Sep 17 18:16:30.286108 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.286092    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.288595 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.288568    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:16:30.288708 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.288636    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-b200d5-openshift-default-dockercfg-lb8wc\""
Sep 17 18:16:30.296627 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.296606    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"]
Sep 17 18:16:30.311055 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.311032    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:30.413389 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.413360    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-credential-socket\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.413573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.413398    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-podinfo\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.413573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.413435    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-envoy\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.413573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.413469    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lgsv\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-kube-api-access-4lgsv\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.413573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.413521    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-token\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.413573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.413558    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istiod-ca-cert\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.413812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.413575    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-socket\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.413812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.413670    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-certs\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.413812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.413709    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-data\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.514996 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.514960    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-envoy\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.514996 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.514993    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lgsv\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-kube-api-access-4lgsv\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515042    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-token\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515075    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istiod-ca-cert\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515092    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-socket\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515135    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-certs\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515160    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-data\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515192    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-credential-socket\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515214    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-podinfo\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515563    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-certs\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515609    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-data\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515828 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515646    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-credential-socket\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515828 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515779    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-socket\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.515828 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.515828    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istiod-ca-cert\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.517433 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.517411    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-envoy\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.517719 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.517700    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-podinfo\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.522766 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.522746    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-token\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.522899 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.522881    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lgsv\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-kube-api-access-4lgsv\") pod \"e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") " pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.596105 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.596020    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:30.732514 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.732467    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:30.871457 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.871374    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"]
Sep 17 18:16:30.874341 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:16:30.874302    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ef3f16c_6cff_4b3c_bb2c_7f61196288ca.slice/crio-5f097bdfaf50ed147ac30a473c8fc02df157ced48054fb97626cdbdb5f7893b2 WatchSource:0}: Error finding container 5f097bdfaf50ed147ac30a473c8fc02df157ced48054fb97626cdbdb5f7893b2: Status 404 returned error can't find the container with id 5f097bdfaf50ed147ac30a473c8fc02df157ced48054fb97626cdbdb5f7893b2
Sep 17 18:16:30.919191 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.919158    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" event={"ID":"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca","Type":"ContainerStarted","Data":"5f097bdfaf50ed147ac30a473c8fc02df157ced48054fb97626cdbdb5f7893b2"}
Sep 17 18:16:30.919521 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:30.919237    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7640-961c34" podUID="2cf05ba8-4543-45c0-ac02-1120b14b19fe" containerName="test-curl-7640-961c34" containerID="cri-o://a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f" gracePeriod=2
Sep 17 18:16:32.031976 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:32.031875    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:32.142692 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:32.142648    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:32.237045 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:32.236828    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:33.038684 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.038615    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:33.803088 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.803033    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:16:33.803175 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.803145    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:16:33.803209 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.803188    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:16:33.883948 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.883923    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7640-961c34"
Sep 17 18:16:33.941799 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.941746    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" event={"ID":"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca","Type":"ContainerStarted","Data":"cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61"}
Sep 17 18:16:33.943157 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.943123    2589 generic.go:356] "Generic (PLEG): container finished" podID="2cf05ba8-4543-45c0-ac02-1120b14b19fe" containerID="a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f" exitCode=137
Sep 17 18:16:33.943290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.943175    2589 scope.go:117] "RemoveContainer" containerID="a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f"
Sep 17 18:16:33.943290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.943171    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7640-961c34"
Sep 17 18:16:33.946351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.946326    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjrfg\" (UniqueName: \"kubernetes.io/projected/2cf05ba8-4543-45c0-ac02-1120b14b19fe-kube-api-access-bjrfg\") pod \"2cf05ba8-4543-45c0-ac02-1120b14b19fe\" (UID: \"2cf05ba8-4543-45c0-ac02-1120b14b19fe\") "
Sep 17 18:16:33.948860 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.948836    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cf05ba8-4543-45c0-ac02-1120b14b19fe-kube-api-access-bjrfg" (OuterVolumeSpecName: "kube-api-access-bjrfg") pod "2cf05ba8-4543-45c0-ac02-1120b14b19fe" (UID: "2cf05ba8-4543-45c0-ac02-1120b14b19fe"). InnerVolumeSpecName "kube-api-access-bjrfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:16:33.952948 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.952927    2589 scope.go:117] "RemoveContainer" containerID="a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f"
Sep 17 18:16:33.953267 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:16:33.953242    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f\": container with ID starting with a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f not found: ID does not exist" containerID="a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f"
Sep 17 18:16:33.953360 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.953277    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f"} err="failed to get container status \"a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f\": rpc error: code = NotFound desc = could not find container \"a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f\": container with ID starting with a8e4b9a678c0177f290772d2c0cf925af33fe19cc750009e25c29af950b5073f not found: ID does not exist"
Sep 17 18:16:33.966549 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:33.966508    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" podStartSLOduration=1.040583903 podStartE2EDuration="3.966495948s" totalImagesPullingTime="2.925912045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:33.965221014 +0000 UTC m=+829.644906870" watchObservedRunningTime="2026-09-17 18:16:33.966495948 +0000 UTC m=+829.646181795"
Sep 17 18:16:34.004930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:34.004900    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:34.048133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:34.048095    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjrfg\" (UniqueName: \"kubernetes.io/projected/2cf05ba8-4543-45c0-ac02-1120b14b19fe-kube-api-access-bjrfg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:16:34.253122 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:16:34.253090    2589 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7640-961c34\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="2cf05ba8-4543-45c0-ac02-1120b14b19fe" pod="opendatahub/test-curl-7640-961c34"
Sep 17 18:16:34.596813 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:34.596779    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:34.601820 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:34.601794    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:34.623353 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:16:34.623324    2589 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7640-961c34\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="2cf05ba8-4543-45c0-ac02-1120b14b19fe" pod="opendatahub/test-curl-7640-961c34"
Sep 17 18:16:34.683631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:34.683603    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2cf05ba8-4543-45c0-ac02-1120b14b19fe" path="/var/lib/kubelet/pods/2cf05ba8-4543-45c0-ac02-1120b14b19fe/volumes"
Sep 17 18:16:34.947618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:34.947535    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:34.949337 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:34.949311    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:34.950159 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:34.950145    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:16:35.046802 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.046769    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:35.296270 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.296241    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:35.517443 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.517404    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:35.583731 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.583651    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:35.617538 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.617501    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"]
Sep 17 18:16:35.622147 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.622130    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:35.624470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.624446    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-015dd779\""
Sep 17 18:16:35.630926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.630902    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"]
Sep 17 18:16:35.636277 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.636258    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"]
Sep 17 18:16:35.639386 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.639367    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:35.641568 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.641549    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-015dd779-dockercfg-db64r\""
Sep 17 18:16:35.650492 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.650472    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"]
Sep 17 18:16:35.763182 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.763154    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6nh4\" (UniqueName: \"kubernetes.io/projected/b311edb6-e2bd-41c7-ab5d-96f2462102b9-kube-api-access-b6nh4\") pod \"payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n\" (UID: \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:35.763331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.763204    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2752cac1-211a-444b-9a6e-ae7f4b34ce25-plugins-config-volume\") pod \"payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd\" (UID: \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:35.763331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.763262    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h5jl\" (UniqueName: \"kubernetes.io/projected/2752cac1-211a-444b-9a6e-ae7f4b34ce25-kube-api-access-8h5jl\") pod \"payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd\" (UID: \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:35.763414 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.763350    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b311edb6-e2bd-41c7-ab5d-96f2462102b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n\" (UID: \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:35.864470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.864407    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h5jl\" (UniqueName: \"kubernetes.io/projected/2752cac1-211a-444b-9a6e-ae7f4b34ce25-kube-api-access-8h5jl\") pod \"payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd\" (UID: \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:35.864470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.864461    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b311edb6-e2bd-41c7-ab5d-96f2462102b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n\" (UID: \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:35.864619 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.864496    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6nh4\" (UniqueName: \"kubernetes.io/projected/b311edb6-e2bd-41c7-ab5d-96f2462102b9-kube-api-access-b6nh4\") pod \"payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n\" (UID: \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:35.864619 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.864524    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2752cac1-211a-444b-9a6e-ae7f4b34ce25-plugins-config-volume\") pod \"payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd\" (UID: \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:35.865469 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.865442    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2752cac1-211a-444b-9a6e-ae7f4b34ce25-plugins-config-volume\") pod \"payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd\" (UID: \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:35.865541 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.865448    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b311edb6-e2bd-41c7-ab5d-96f2462102b9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n\" (UID: \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:35.872312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.872291    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6nh4\" (UniqueName: \"kubernetes.io/projected/b311edb6-e2bd-41c7-ab5d-96f2462102b9-kube-api-access-b6nh4\") pod \"payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n\" (UID: \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:35.872414 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.872326    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h5jl\" (UniqueName: \"kubernetes.io/projected/2752cac1-211a-444b-9a6e-ae7f4b34ce25-kube-api-access-8h5jl\") pod \"payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd\" (UID: \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\") " pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:35.928988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.928957    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:35.932465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.932450    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:35.948939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:35.948920    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:36.034721 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:36.034672    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:36.114155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:36.114119    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:36.213126 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:36.213096    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"]
Sep 17 18:16:36.214677 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:16:36.214649    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb311edb6_e2bd_41c7_ab5d_96f2462102b9.slice/crio-22ec1247c6734d365e4d8d1856713b0dd849066ec4f1e0df13ec69f9623b14c7 WatchSource:0}: Error finding container 22ec1247c6734d365e4d8d1856713b0dd849066ec4f1e0df13ec69f9623b14c7: Status 404 returned error can't find the container with id 22ec1247c6734d365e4d8d1856713b0dd849066ec4f1e0df13ec69f9623b14c7
Sep 17 18:16:36.236234 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:36.236212    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"]
Sep 17 18:16:36.237989 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:16:36.237966    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2752cac1_211a_444b_9a6e_ae7f4b34ce25.slice/crio-d961986f9ff0413c642a3c3fb65abd9d74a306b4f55740452a7b8f9b0ba11c7d WatchSource:0}: Error finding container d961986f9ff0413c642a3c3fb65abd9d74a306b4f55740452a7b8f9b0ba11c7d: Status 404 returned error can't find the container with id d961986f9ff0413c642a3c3fb65abd9d74a306b4f55740452a7b8f9b0ba11c7d
Sep 17 18:16:36.860037 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:36.859965    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:36.961744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:36.961699    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n" event={"ID":"b311edb6-e2bd-41c7-ab5d-96f2462102b9","Type":"ContainerStarted","Data":"22ec1247c6734d365e4d8d1856713b0dd849066ec4f1e0df13ec69f9623b14c7"}
Sep 17 18:16:36.963373 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:36.963346    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd" event={"ID":"2752cac1-211a-444b-9a6e-ae7f4b34ce25","Type":"ContainerStarted","Data":"d961986f9ff0413c642a3c3fb65abd9d74a306b4f55740452a7b8f9b0ba11c7d"}
Sep 17 18:16:37.359760 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:37.359727    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:37.459774 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:37.459746    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:37.722589 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:37.722498    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:37.831617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:37.831564    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:37.928376 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:37.928346    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:37.970063 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:37.970030    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd" event={"ID":"2752cac1-211a-444b-9a6e-ae7f4b34ce25","Type":"ContainerStarted","Data":"6aa6ac0c7be97921916f24fa85231b4b18ca20da2e3b63c0833991938cb7c937"}
Sep 17 18:16:37.970218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:37.970100    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:37.972207 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:37.972182    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n" event={"ID":"b311edb6-e2bd-41c7-ab5d-96f2462102b9","Type":"ContainerStarted","Data":"e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb"}
Sep 17 18:16:37.972358 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:37.972339    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:38.013305 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:38.013262    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd" podStartSLOduration=1.8140634740000001 podStartE2EDuration="3.013245187s" totalImagesPullingTime="1.199181713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:38.011088616 +0000 UTC m=+833.690774471" watchObservedRunningTime="2026-09-17 18:16:38.013245187 +0000 UTC m=+833.692931040"
Sep 17 18:16:38.030025 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:38.029958    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n" podStartSLOduration=1.809944504 podStartE2EDuration="3.029938765s" totalImagesPullingTime="1.219994261s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:38.02973112 +0000 UTC m=+833.709416976" watchObservedRunningTime="2026-09-17 18:16:38.029938765 +0000 UTC m=+833.709624627"
Sep 17 18:16:38.045474 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:38.045444    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:38.140466 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:38.140437    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:38.754256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:38.754221    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:39.527798 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:39.527761    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:39.627834 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:39.627782    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:39.973454 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:39.973360    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:40.069342 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.069306    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:40.114526 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.114496    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"]
Sep 17 18:16:40.117821 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.117805    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:40.120077 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.120055    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-b200d5\""
Sep 17 18:16:40.129772 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.129752    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"]
Sep 17 18:16:40.138827 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.138806    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:40.156496 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.156470    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"]
Sep 17 18:16:40.159669 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.159654    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:40.162155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.162139    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-b200d5-dockercfg-nv7fh\""
Sep 17 18:16:40.173647 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.173621    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"]
Sep 17 18:16:40.205962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.205941    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pb2bm\" (UniqueName: \"kubernetes.io/projected/8f9bf969-372f-42f4-b7e5-f99acde0d955-kube-api-access-pb2bm\") pod \"payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2\" (UID: \"8f9bf969-372f-42f4-b7e5-f99acde0d955\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:40.206079 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.205975    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9bf969-372f-42f4-b7e5-f99acde0d955-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2\" (UID: \"8f9bf969-372f-42f4-b7e5-f99acde0d955\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:40.306776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.306736    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d678436-9496-4e8f-b87e-6c5975966f44-plugins-config-volume\") pod \"payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8\" (UID: \"1d678436-9496-4e8f-b87e-6c5975966f44\") " pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:40.307057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.306792    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pb2bm\" (UniqueName: \"kubernetes.io/projected/8f9bf969-372f-42f4-b7e5-f99acde0d955-kube-api-access-pb2bm\") pod \"payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2\" (UID: \"8f9bf969-372f-42f4-b7e5-f99acde0d955\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:40.307057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.306821    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9bf969-372f-42f4-b7e5-f99acde0d955-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2\" (UID: \"8f9bf969-372f-42f4-b7e5-f99acde0d955\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:40.307057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.306868    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s59mq\" (UniqueName: \"kubernetes.io/projected/1d678436-9496-4e8f-b87e-6c5975966f44-kube-api-access-s59mq\") pod \"payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8\" (UID: \"1d678436-9496-4e8f-b87e-6c5975966f44\") " pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:40.307669 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.307643    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9bf969-372f-42f4-b7e5-f99acde0d955-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2\" (UID: \"8f9bf969-372f-42f4-b7e5-f99acde0d955\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:40.314687 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.314660    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pb2bm\" (UniqueName: \"kubernetes.io/projected/8f9bf969-372f-42f4-b7e5-f99acde0d955-kube-api-access-pb2bm\") pod \"payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2\" (UID: \"8f9bf969-372f-42f4-b7e5-f99acde0d955\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:40.408216 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.408188    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s59mq\" (UniqueName: \"kubernetes.io/projected/1d678436-9496-4e8f-b87e-6c5975966f44-kube-api-access-s59mq\") pod \"payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8\" (UID: \"1d678436-9496-4e8f-b87e-6c5975966f44\") " pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:40.408318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.408257    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d678436-9496-4e8f-b87e-6c5975966f44-plugins-config-volume\") pod \"payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8\" (UID: \"1d678436-9496-4e8f-b87e-6c5975966f44\") " pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:40.408839 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.408821    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d678436-9496-4e8f-b87e-6c5975966f44-plugins-config-volume\") pod \"payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8\" (UID: \"1d678436-9496-4e8f-b87e-6c5975966f44\") " pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:40.416763 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.416737    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s59mq\" (UniqueName: \"kubernetes.io/projected/1d678436-9496-4e8f-b87e-6c5975966f44-kube-api-access-s59mq\") pod \"payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8\" (UID: \"1d678436-9496-4e8f-b87e-6c5975966f44\") " pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:40.427712 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.427693    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:40.429911 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.429889    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:40.470752 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.470717    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:40.701400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.701377    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"]
Sep 17 18:16:40.703234 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:16:40.703210    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f9bf969_372f_42f4_b7e5_f99acde0d955.slice/crio-a3a5fb7d02ba8d7ea745cb317343bc3c73f3e3ed1042cb3af5d1c650b00cc934 WatchSource:0}: Error finding container a3a5fb7d02ba8d7ea745cb317343bc3c73f3e3ed1042cb3af5d1c650b00cc934: Status 404 returned error can't find the container with id a3a5fb7d02ba8d7ea745cb317343bc3c73f3e3ed1042cb3af5d1c650b00cc934
Sep 17 18:16:40.749136 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.749113    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"]
Sep 17 18:16:40.750773 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:16:40.750745    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d678436_9496_4e8f_b87e_6c5975966f44.slice/crio-e33d49a9a68a60d391e0f82f3050d0c510b8c6aed6330484c0cb14768d9062c2 WatchSource:0}: Error finding container e33d49a9a68a60d391e0f82f3050d0c510b8c6aed6330484c0cb14768d9062c2: Status 404 returned error can't find the container with id e33d49a9a68a60d391e0f82f3050d0c510b8c6aed6330484c0cb14768d9062c2
Sep 17 18:16:40.988161 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.988069    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2" event={"ID":"8f9bf969-372f-42f4-b7e5-f99acde0d955","Type":"ContainerStarted","Data":"0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02"}
Sep 17 18:16:40.988161 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.988131    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2" event={"ID":"8f9bf969-372f-42f4-b7e5-f99acde0d955","Type":"ContainerStarted","Data":"a3a5fb7d02ba8d7ea745cb317343bc3c73f3e3ed1042cb3af5d1c650b00cc934"}
Sep 17 18:16:40.988161 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.988158    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:40.989922 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.989896    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8" event={"ID":"1d678436-9496-4e8f-b87e-6c5975966f44","Type":"ContainerStarted","Data":"af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a"}
Sep 17 18:16:40.990067 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.989945    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8" event={"ID":"1d678436-9496-4e8f-b87e-6c5975966f44","Type":"ContainerStarted","Data":"e33d49a9a68a60d391e0f82f3050d0c510b8c6aed6330484c0cb14768d9062c2"}
Sep 17 18:16:40.990067 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:40.989973    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:41.005625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:41.005582    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2" podStartSLOduration=1.005569885 podStartE2EDuration="1.005569885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:41.004035313 +0000 UTC m=+836.683721164" watchObservedRunningTime="2026-09-17 18:16:41.005569885 +0000 UTC m=+836.685255791"
Sep 17 18:16:41.021910 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:41.021874    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8" podStartSLOduration=1.021862842 podStartE2EDuration="1.021862842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:41.020846832 +0000 UTC m=+836.700532687" watchObservedRunningTime="2026-09-17 18:16:41.021862842 +0000 UTC m=+836.701548766"
Sep 17 18:16:41.073240 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:41.073212    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:41.218868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:41.218830    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:41.357900 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:41.357869    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:41.514678 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:41.514640    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:41.616421 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:41.616344    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:41.709319 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:41.709288    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:16:42.144663 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:42.144616    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:42.233160 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:42.233114    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:42.453878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:42.453790    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:42.549636 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:42.549600    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:42.927464 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:42.927435    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:43.017844 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:43.017804    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:43.118201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:43.118167    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:43.344287 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:43.344235    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:44.071807 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:44.071775    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:44.450655 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:44.450565    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:45.152533 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:45.152500    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:45.953673 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:45.953637    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:46.021716 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.021685    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:46.541377 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.541337    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:46.639187 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.639156    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:46.664246 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.664213    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"]
Sep 17 18:16:46.668783 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.668762    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:46.671267 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.671250    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-99ac10\""
Sep 17 18:16:46.671372 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.671282    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-99ac10-dockercfg-qtwls\""
Sep 17 18:16:46.684323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.684303    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"]
Sep 17 18:16:46.865798 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.865730    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c51caf-eacb-4774-945f-87c518c66e8d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd\" (UID: \"78c51caf-eacb-4774-945f-87c518c66e8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:46.865962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.865896    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62mxg\" (UniqueName: \"kubernetes.io/projected/78c51caf-eacb-4774-945f-87c518c66e8d-kube-api-access-62mxg\") pod \"payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd\" (UID: \"78c51caf-eacb-4774-945f-87c518c66e8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:46.942609 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.942583    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:46.967278 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.967249    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c51caf-eacb-4774-945f-87c518c66e8d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd\" (UID: \"78c51caf-eacb-4774-945f-87c518c66e8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:46.967403 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.967300    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62mxg\" (UniqueName: \"kubernetes.io/projected/78c51caf-eacb-4774-945f-87c518c66e8d-kube-api-access-62mxg\") pod \"payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd\" (UID: \"78c51caf-eacb-4774-945f-87c518c66e8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:46.967941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.967923    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c51caf-eacb-4774-945f-87c518c66e8d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd\" (UID: \"78c51caf-eacb-4774-945f-87c518c66e8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:46.975990 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.975966    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62mxg\" (UniqueName: \"kubernetes.io/projected/78c51caf-eacb-4774-945f-87c518c66e8d-kube-api-access-62mxg\") pod \"payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd\" (UID: \"78c51caf-eacb-4774-945f-87c518c66e8d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:46.977815 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:46.977799    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:47.042329 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:47.042298    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:47.248745 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:47.248721    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"]
Sep 17 18:16:47.251402 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:16:47.251374    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78c51caf_eacb_4774_945f_87c518c66e8d.slice/crio-4597200aa1faef76684cf892e38b253e0e85a039dc03622114809e8c5318bffe WatchSource:0}: Error finding container 4597200aa1faef76684cf892e38b253e0e85a039dc03622114809e8c5318bffe: Status 404 returned error can't find the container with id 4597200aa1faef76684cf892e38b253e0e85a039dc03622114809e8c5318bffe
Sep 17 18:16:47.960322 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:47.960287    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:48.024805 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:48.024772    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd" event={"ID":"78c51caf-eacb-4774-945f-87c518c66e8d","Type":"ContainerStarted","Data":"203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5"}
Sep 17 18:16:48.024945 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:48.024829    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd" event={"ID":"78c51caf-eacb-4774-945f-87c518c66e8d","Type":"ContainerStarted","Data":"4597200aa1faef76684cf892e38b253e0e85a039dc03622114809e8c5318bffe"}
Sep 17 18:16:48.024945 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:48.024874    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:48.041826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:48.041777    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd" podStartSLOduration=2.041761813 podStartE2EDuration="2.041761813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:16:48.041258869 +0000 UTC m=+843.720944724" watchObservedRunningTime="2026-09-17 18:16:48.041761813 +0000 UTC m=+843.721447667"
Sep 17 18:16:48.696244 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:48.696206    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:48.811351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:48.811311    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:48.911152 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:48.911123    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:48.978213 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:48.978131    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:16:48.978593 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:48.978388    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:16:49.027459 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:49.027424    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:49.553947 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:49.553912    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:49.651493 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:49.651466    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:50.628727 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:50.628685    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:51.461328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:51.461296    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:52.000243 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:52.000205    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:16:52.001183 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:52.001159    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:16:52.230758 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:52.230719    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:53.238310 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:53.238280    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:54.155837 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:54.155802    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:54.745068 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:54.745038    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:55.109832 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:55.109796    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:55.627813 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:55.627774    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:55.853508 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:55.853470    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:55.968195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:55.968117    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:56.071195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:56.071165    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:56.165765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:56.165734    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:56.268361 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:56.268334    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:56.972793 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:56.972765    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:57.746960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:57.746910    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:58.659125 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:58.659091    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:16:59.032489 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:59.032463    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:16:59.539557 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:16:59.539518    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:00.299147 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:00.299110    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:17:00.304303 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:00.304277    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:01.052878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:01.052839    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:01.101531 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:01.101498    2589 generic.go:356] "Generic (PLEG): container finished" podID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerID="cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61" exitCode=0
Sep 17 18:17:01.101531 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:01.101534    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" event={"ID":"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca","Type":"ContainerDied","Data":"cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61"}
Sep 17 18:17:01.101920 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:01.101905    2589 scope.go:117] "RemoveContainer" containerID="cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61"
Sep 17 18:17:01.103142 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:01.103104    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:17:01.103253 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:01.103196    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:17:01.103316 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:01.103247    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:17:01.821954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:01.821920    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:02.037327 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:02.037290    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:02.109839 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:02.109746    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" event={"ID":"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca","Type":"ContainerStarted","Data":"3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b"}
Sep 17 18:17:02.597124 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:02.597026    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:17:02.598766 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:02.598426    2589 patch_prober.go:28] interesting pod/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 17 18:17:02.598766 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:02.598500    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 17 18:17:02.670312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:02.670271    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:03.173319 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:03.173271    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:03.242290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:03.242256    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:03.564403 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:03.564363    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:03.596810 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:03.596778    2589 patch_prober.go:28] interesting pod/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 17 18:17:03.597097 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:03.597067    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 17 18:17:04.346021 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:04.345973    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:04.596943 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:04.596833    2589 patch_prober.go:28] interesting pod/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 17 18:17:04.596943 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:04.596915    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 17 18:17:04.949788 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:04.949697    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:17:04.964650 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:04.964621    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:04.996354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:04.996063    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"]
Sep 17 18:17:04.996354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:04.996300    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd" podUID="2752cac1-211a-444b-9a6e-ae7f4b34ce25" containerName="payload-processing" containerID="cri-o://6aa6ac0c7be97921916f24fa85231b4b18ca20da2e3b63c0833991938cb7c937" gracePeriod=30
Sep 17 18:17:05.005289 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.005255    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"]
Sep 17 18:17:05.005550 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.005512    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n" podUID="b311edb6-e2bd-41c7-ab5d-96f2462102b9" containerName="payload-pre-processing" containerID="cri-o://e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb" gracePeriod=30
Sep 17 18:17:05.038025 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.037983    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:05.125192 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.125168    2589 generic.go:356] "Generic (PLEG): container finished" podID="2752cac1-211a-444b-9a6e-ae7f4b34ce25" containerID="6aa6ac0c7be97921916f24fa85231b4b18ca20da2e3b63c0833991938cb7c937" exitCode=0
Sep 17 18:17:05.125292 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.125260    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd" event={"ID":"2752cac1-211a-444b-9a6e-ae7f4b34ce25","Type":"ContainerDied","Data":"6aa6ac0c7be97921916f24fa85231b4b18ca20da2e3b63c0833991938cb7c937"}
Sep 17 18:17:05.140644 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.140599    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:05.221481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.221406    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:05.264177 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.264156    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:17:05.316647 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.316563    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:17:05.325724 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.325700    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b311edb6-e2bd-41c7-ab5d-96f2462102b9-plugins-config-volume\") pod \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\" (UID: \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\") "
Sep 17 18:17:05.325891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.325733    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h5jl\" (UniqueName: \"kubernetes.io/projected/2752cac1-211a-444b-9a6e-ae7f4b34ce25-kube-api-access-8h5jl\") pod \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\" (UID: \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\") "
Sep 17 18:17:05.325891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.325784    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2752cac1-211a-444b-9a6e-ae7f4b34ce25-plugins-config-volume\") pod \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\" (UID: \"2752cac1-211a-444b-9a6e-ae7f4b34ce25\") "
Sep 17 18:17:05.325891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.325812    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6nh4\" (UniqueName: \"kubernetes.io/projected/b311edb6-e2bd-41c7-ab5d-96f2462102b9-kube-api-access-b6nh4\") pod \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\" (UID: \"b311edb6-e2bd-41c7-ab5d-96f2462102b9\") "
Sep 17 18:17:05.326224 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.326175    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b311edb6-e2bd-41c7-ab5d-96f2462102b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b311edb6-e2bd-41c7-ab5d-96f2462102b9" (UID: "b311edb6-e2bd-41c7-ab5d-96f2462102b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:05.326481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.326443    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2752cac1-211a-444b-9a6e-ae7f4b34ce25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2752cac1-211a-444b-9a6e-ae7f4b34ce25" (UID: "2752cac1-211a-444b-9a6e-ae7f4b34ce25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:05.330167 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.330141    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2752cac1-211a-444b-9a6e-ae7f4b34ce25-kube-api-access-8h5jl" (OuterVolumeSpecName: "kube-api-access-8h5jl") pod "2752cac1-211a-444b-9a6e-ae7f4b34ce25" (UID: "2752cac1-211a-444b-9a6e-ae7f4b34ce25"). InnerVolumeSpecName "kube-api-access-8h5jl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:05.330332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.330303    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b311edb6-e2bd-41c7-ab5d-96f2462102b9-kube-api-access-b6nh4" (OuterVolumeSpecName: "kube-api-access-b6nh4") pod "b311edb6-e2bd-41c7-ab5d-96f2462102b9" (UID: "b311edb6-e2bd-41c7-ab5d-96f2462102b9"). InnerVolumeSpecName "kube-api-access-b6nh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:05.375381 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.375346    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:05.426485 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.426428    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2752cac1-211a-444b-9a6e-ae7f4b34ce25-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:05.426485 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.426481    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6nh4\" (UniqueName: \"kubernetes.io/projected/b311edb6-e2bd-41c7-ab5d-96f2462102b9-kube-api-access-b6nh4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:05.426704 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.426511    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b311edb6-e2bd-41c7-ab5d-96f2462102b9-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:05.426704 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.426523    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h5jl\" (UniqueName: \"kubernetes.io/projected/2752cac1-211a-444b-9a6e-ae7f4b34ce25-kube-api-access-8h5jl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:05.596677 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.596631    2589 patch_prober.go:28] interesting pod/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 17 18:17:05.596893 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.596700    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 17 18:17:05.790903 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.790860    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:05.903041 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:05.902920    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:06.046899 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.046865    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:06.131419 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.131383    2589 generic.go:356] "Generic (PLEG): container finished" podID="b311edb6-e2bd-41c7-ab5d-96f2462102b9" containerID="e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb" exitCode=0
Sep 17 18:17:06.131607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.131425    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"
Sep 17 18:17:06.131607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.131464    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n" event={"ID":"b311edb6-e2bd-41c7-ab5d-96f2462102b9","Type":"ContainerDied","Data":"e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb"}
Sep 17 18:17:06.131607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.131511    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n" event={"ID":"b311edb6-e2bd-41c7-ab5d-96f2462102b9","Type":"ContainerDied","Data":"22ec1247c6734d365e4d8d1856713b0dd849066ec4f1e0df13ec69f9623b14c7"}
Sep 17 18:17:06.131607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.131533    2589 scope.go:117] "RemoveContainer" containerID="e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb"
Sep 17 18:17:06.132950 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.132934    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd" event={"ID":"2752cac1-211a-444b-9a6e-ae7f4b34ce25","Type":"ContainerDied","Data":"d961986f9ff0413c642a3c3fb65abd9d74a306b4f55740452a7b8f9b0ba11c7d"}
Sep 17 18:17:06.133051 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.132999    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"
Sep 17 18:17:06.141742 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.141726    2589 scope.go:117] "RemoveContainer" containerID="e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb"
Sep 17 18:17:06.142057 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:17:06.142036    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb\": container with ID starting with e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb not found: ID does not exist" containerID="e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb"
Sep 17 18:17:06.142140 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.142063    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb"} err="failed to get container status \"e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb\": rpc error: code = NotFound desc = could not find container \"e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb\": container with ID starting with e2282b56a069d15250272bde61220126b0c5be47f1f9847b531761e890de8dbb not found: ID does not exist"
Sep 17 18:17:06.142140 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.142078    2589 scope.go:117] "RemoveContainer" containerID="6aa6ac0c7be97921916f24fa85231b4b18ca20da2e3b63c0833991938cb7c937"
Sep 17 18:17:06.161029 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.160957    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"]
Sep 17 18:17:06.164197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.164174    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-015dd779-7857cdcf7d-88d5n"]
Sep 17 18:17:06.177947 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.177928    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"]
Sep 17 18:17:06.189299 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.189270    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-015dd779-5886fb7969-hwgdd"]
Sep 17 18:17:06.341926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.341897    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"]
Sep 17 18:17:06.342350 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.342332    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b311edb6-e2bd-41c7-ab5d-96f2462102b9" containerName="payload-pre-processing"
Sep 17 18:17:06.342467 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.342352    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="b311edb6-e2bd-41c7-ab5d-96f2462102b9" containerName="payload-pre-processing"
Sep 17 18:17:06.342467 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.342387    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2752cac1-211a-444b-9a6e-ae7f4b34ce25" containerName="payload-processing"
Sep 17 18:17:06.342467 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.342396    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="2752cac1-211a-444b-9a6e-ae7f4b34ce25" containerName="payload-processing"
Sep 17 18:17:06.342628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.342490    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="2752cac1-211a-444b-9a6e-ae7f4b34ce25" containerName="payload-processing"
Sep 17 18:17:06.342628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.342506    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="b311edb6-e2bd-41c7-ab5d-96f2462102b9" containerName="payload-pre-processing"
Sep 17 18:17:06.345510 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.345490    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.347969 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.347952    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4390f8ca-gw-openshift-default-dockercfg-wfznb\""
Sep 17 18:17:06.361827 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.361806    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"]
Sep 17 18:17:06.393030 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.387950    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:06.433540 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.433448    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-certs\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.433540 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.433494    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-credential-socket\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.433765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.433558    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhfk\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-kube-api-access-jzhfk\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.433765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.433584    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-token\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.433765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.433616    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istiod-ca-cert\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.433765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.433668    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-data\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.433765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.433731    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-socket\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.433962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.433768    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-envoy\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.433962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.433787    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-podinfo\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.534949 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.534913    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-certs\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.534949 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.534951    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-credential-socket\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.534973    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhfk\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-kube-api-access-jzhfk\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.534992    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-token\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535021    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istiod-ca-cert\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535071    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-data\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535105    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-socket\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535139    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-envoy\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535161    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-podinfo\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535574 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535414    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-certs\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535574 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535475    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-credential-socket\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535688 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535598    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-data\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535749 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535701    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istiod-ca-cert\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.535866 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.535830    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-socket\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.537702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.537667    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-envoy\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.537802 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.537784    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-podinfo\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.544080 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.544054    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-token\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.544355 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.544336    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhfk\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-kube-api-access-jzhfk\") pod \"e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") " pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.597616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.597176    2589 patch_prober.go:28] interesting pod/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused" start-of-body=
Sep 17 18:17:06.597616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.597237    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.34:15021/healthz/ready\": dial tcp 10.132.0.34:15021: connect: connection refused"
Sep 17 18:17:06.655298 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.655260    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:06.708696 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.708603    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2752cac1-211a-444b-9a6e-ae7f4b34ce25" path="/var/lib/kubelet/pods/2752cac1-211a-444b-9a6e-ae7f4b34ce25/volumes"
Sep 17 18:17:06.710049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.709269    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b311edb6-e2bd-41c7-ab5d-96f2462102b9" path="/var/lib/kubelet/pods/b311edb6-e2bd-41c7-ab5d-96f2462102b9/volumes"
Sep 17 18:17:06.750668 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.750586    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:06.904999 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.904956    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:06.955574 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.955550    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"]
Sep 17 18:17:06.957597 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:17:06.957569    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd491ac7_9aa5_4d43_b8cc_e2fad1165450.slice/crio-97486b1f448bc5c8f044fa376056c463c3c8568534b9e72e34e4408d93f24b39 WatchSource:0}: Error finding container 97486b1f448bc5c8f044fa376056c463c3c8568534b9e72e34e4408d93f24b39: Status 404 returned error can't find the container with id 97486b1f448bc5c8f044fa376056c463c3c8568534b9e72e34e4408d93f24b39
Sep 17 18:17:06.959865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.959823    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:17:06.959941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.959899    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:17:06.959941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:06.959934    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:17:07.139870 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:07.139832    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" event={"ID":"dd491ac7-9aa5-4d43-b8cc-e2fad1165450","Type":"ContainerStarted","Data":"2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993"}
Sep 17 18:17:07.140043 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:07.139900    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" event={"ID":"dd491ac7-9aa5-4d43-b8cc-e2fad1165450","Type":"ContainerStarted","Data":"97486b1f448bc5c8f044fa376056c463c3c8568534b9e72e34e4408d93f24b39"}
Sep 17 18:17:07.162015 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:07.161946    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" podStartSLOduration=1.161928079 podStartE2EDuration="1.161928079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:07.159964153 +0000 UTC m=+862.839650032" watchObservedRunningTime="2026-09-17 18:17:07.161928079 +0000 UTC m=+862.841613933"
Sep 17 18:17:07.600539 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:07.600512    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:17:07.601489 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:07.601473    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:17:07.630522 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:07.630495    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:07.656366 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:07.656328    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:08.533466 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:08.533430    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:08.661176 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:08.661149    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:08.692361 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:08.692330    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:08.759876 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:08.759841    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:09.150765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:09.150735    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:09.151654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:09.151637    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:09.324734 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:09.324689    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:10.032778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:10.032745    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:10.991814 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:10.991773    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:11.099101 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:11.099067    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:11.570935 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:11.570894    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:11.668448 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:11.668406    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:12.197376 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:12.197334    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:13.006438 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:13.006392    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:14.143921 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:14.143893    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:14.845111 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:14.845080    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:18.503088 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:18.503046    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:17:27.880071 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:27.879972    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"]
Sep 17 18:17:27.883014 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:27.882987    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:27.885541 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:27.885518    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"openshift-service-ca.crt\""
Sep 17 18:17:27.886528 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:27.886503    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"simulator-w4-b18433-kserve-self-signed-certs\""
Sep 17 18:17:27.886636 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:27.886509    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"default-dockercfg-2nzcz\""
Sep 17 18:17:27.886636 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:27.886508    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"kube-root-ca.crt\""
Sep 17 18:17:27.892720 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:27.892693    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"]
Sep 17 18:17:28.020620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.020578    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b08501b-8774-407d-ae48-32db6c7ca4c0-tls-certs\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.020816 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.020697    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-tmp-dir\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.020816 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.020741    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfrk4\" (UniqueName: \"kubernetes.io/projected/7b08501b-8774-407d-ae48-32db6c7ca4c0-kube-api-access-nfrk4\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.020949 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.020831    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-model-cache\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.020949 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.020895    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-home\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.021088 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.020948    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-dshm\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.122161 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.122119    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-tmp-dir\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.123532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.122170    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfrk4\" (UniqueName: \"kubernetes.io/projected/7b08501b-8774-407d-ae48-32db6c7ca4c0-kube-api-access-nfrk4\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.123532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.122216    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-model-cache\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.123532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.122258    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-home\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.123532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.122301    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-dshm\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.123532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.122350    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b08501b-8774-407d-ae48-32db6c7ca4c0-tls-certs\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.123532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.122593    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-tmp-dir\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.123532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.122640    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-model-cache\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.123532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.122667    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-home\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.124817 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.124789    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-dshm\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.125041 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.125022    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b08501b-8774-407d-ae48-32db6c7ca4c0-tls-certs\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.131084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.131039    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfrk4\" (UniqueName: \"kubernetes.io/projected/7b08501b-8774-407d-ae48-32db6c7ca4c0-kube-api-access-nfrk4\") pod \"simulator-w4-b18433-kserve-855849ffb4-5lq9m\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") " pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.192960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.192921    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:28.258936 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.258893    2589 generic.go:356] "Generic (PLEG): container finished" podID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerID="2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993" exitCode=0
Sep 17 18:17:28.259107 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.258983    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" event={"ID":"dd491ac7-9aa5-4d43-b8cc-e2fad1165450","Type":"ContainerDied","Data":"2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993"}
Sep 17 18:17:28.259911 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.259510    2589 scope.go:117] "RemoveContainer" containerID="2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993"
Sep 17 18:17:28.261315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.261037    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:17:28.261315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.261126    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:17:28.261315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.261159    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:17:28.292036 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.291974    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:28.464325 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:28.464201    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"]
Sep 17 18:17:28.466718 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:17:28.466689    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7b08501b_8774_407d_ae48_32db6c7ca4c0.slice/crio-5f1e0818b181433eb29603053296536c5f93036e9be36439567902abc494cc75 WatchSource:0}: Error finding container 5f1e0818b181433eb29603053296536c5f93036e9be36439567902abc494cc75: Status 404 returned error can't find the container with id 5f1e0818b181433eb29603053296536c5f93036e9be36439567902abc494cc75
Sep 17 18:17:29.265666 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:29.265623    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" event={"ID":"dd491ac7-9aa5-4d43-b8cc-e2fad1165450","Type":"ContainerStarted","Data":"d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554"}
Sep 17 18:17:29.267305 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:29.267277    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m" event={"ID":"7b08501b-8774-407d-ae48-32db6c7ca4c0","Type":"ContainerStarted","Data":"20cc916f56811deb3208563afddd102db2b19e61ba8eed6be154d0f10626093b"}
Sep 17 18:17:29.267439 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:29.267315    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m" event={"ID":"7b08501b-8774-407d-ae48-32db6c7ca4c0","Type":"ContainerStarted","Data":"5f1e0818b181433eb29603053296536c5f93036e9be36439567902abc494cc75"}
Sep 17 18:17:29.310966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:29.310911    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m" podStartSLOduration=2.310893685 podStartE2EDuration="2.310893685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:29.308986441 +0000 UTC m=+884.988672297" watchObservedRunningTime="2026-09-17 18:17:29.310893685 +0000 UTC m=+884.990579538"
Sep 17 18:17:29.656117 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:29.655992    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:29.656316 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:29.656297    2589 patch_prober.go:28] interesting pod/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 17 18:17:29.656370 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:29.656344    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 17 18:17:30.656442 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:30.656405    2589 patch_prober.go:28] interesting pod/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 17 18:17:30.656858 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:30.656479    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 17 18:17:31.656253 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:31.656215    2589 patch_prober.go:28] interesting pod/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 17 18:17:31.656464 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:31.656273    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 17 18:17:32.655737 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:32.655704    2589 patch_prober.go:28] interesting pod/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 17 18:17:32.656150 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:32.655771    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 17 18:17:33.092811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:33.092776    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:33.656167 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:33.656135    2589 patch_prober.go:28] interesting pod/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 17 18:17:33.656573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:33.656194    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 17 18:17:34.059486 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:34.059444    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:34.113265 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:34.113223    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:17:34.661132 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:34.661105    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:34.661557 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:34.661389    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:34.668462 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:34.668432    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:34.935939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:34.935843    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:35.834825 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:35.834792    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:36.600399 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:36.600364    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:37.440189 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:37.440155    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:38.193139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:38.193105    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:38.193330 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:38.193158    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:38.205859 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:38.205833    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:38.231088 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:38.231057    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:38.336349 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:38.336318    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:17:38.974643 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:38.974607    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:39.574944 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:39.574906    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:39.652570 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:39.652531    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:39.718456 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:39.718420    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:39.804442 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:39.804411    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:39.864595 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:39.864507    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:39.936516 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:39.936484    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"]
Sep 17 18:17:39.936772 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:39.936747    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy" containerID="cri-o://d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554" gracePeriod=30
Sep 17 18:17:39.975746 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:39.975712    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:40.289306 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:40.289270    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:40.388782 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:40.388729    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:40.676862 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:40.676774    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:40.783138 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:40.783109    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:41.033255 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:41.033223    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:41.801451 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:41.801423    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:42.607423 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:42.607393    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:42.731883 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:42.731852    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:42.832954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:42.832925    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:43.571404 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:43.571369    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:44.322429 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:44.322375    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:45.144320 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.144295    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:45.200876 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.200827    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:45.227202 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.227167    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:45.357169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.357138    2589 generic.go:356] "Generic (PLEG): container finished" podID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerID="d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554" exitCode=0
Sep 17 18:17:45.357547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.357186    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"
Sep 17 18:17:45.357547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.357219    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" event={"ID":"dd491ac7-9aa5-4d43-b8cc-e2fad1165450","Type":"ContainerDied","Data":"d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554"}
Sep 17 18:17:45.357547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.357272    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv" event={"ID":"dd491ac7-9aa5-4d43-b8cc-e2fad1165450","Type":"ContainerDied","Data":"97486b1f448bc5c8f044fa376056c463c3c8568534b9e72e34e4408d93f24b39"}
Sep 17 18:17:45.357547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.357294    2589 scope.go:117] "RemoveContainer" containerID="d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554"
Sep 17 18:17:45.365795 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.365778    2589 scope.go:117] "RemoveContainer" containerID="2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993"
Sep 17 18:17:45.373659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.373644    2589 scope.go:117] "RemoveContainer" containerID="d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554"
Sep 17 18:17:45.373895 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:17:45.373873    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554\": container with ID starting with d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554 not found: ID does not exist" containerID="d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554"
Sep 17 18:17:45.373941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.373907    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554"} err="failed to get container status \"d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554\": rpc error: code = NotFound desc = could not find container \"d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554\": container with ID starting with d512150d7758395305a6162f3f0696ccad03ebe7b9446957e21199efc0387554 not found: ID does not exist"
Sep 17 18:17:45.373941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.373925    2589 scope.go:117] "RemoveContainer" containerID="2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993"
Sep 17 18:17:45.374205 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:17:45.374187    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993\": container with ID starting with 2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993 not found: ID does not exist" containerID="2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993"
Sep 17 18:17:45.374256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.374212    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993"} err="failed to get container status \"2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993\": rpc error: code = NotFound desc = could not find container \"2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993\": container with ID starting with 2fffb77a236a925d183302ce6b9ec9522484f578a750375a07e14093756e4993 not found: ID does not exist"
Sep 17 18:17:45.382509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382491    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-certs\") pod \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") "
Sep 17 18:17:45.382578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382527    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-envoy\") pod \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") "
Sep 17 18:17:45.382578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382548    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istiod-ca-cert\") pod \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") "
Sep 17 18:17:45.382689 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382592    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-credential-socket\") pod \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") "
Sep 17 18:17:45.382689 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382649    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-token\") pod \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") "
Sep 17 18:17:45.382787 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382697    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-data\") pod \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") "
Sep 17 18:17:45.382787 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382751    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-podinfo\") pod \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") "
Sep 17 18:17:45.382787 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382767    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "dd491ac7-9aa5-4d43-b8cc-e2fad1165450" (UID: "dd491ac7-9aa5-4d43-b8cc-e2fad1165450"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:45.382939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382802    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-socket\") pod \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") "
Sep 17 18:17:45.382939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382838    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jzhfk\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-kube-api-access-jzhfk\") pod \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\" (UID: \"dd491ac7-9aa5-4d43-b8cc-e2fad1165450\") "
Sep 17 18:17:45.382939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382906    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "dd491ac7-9aa5-4d43-b8cc-e2fad1165450" (UID: "dd491ac7-9aa5-4d43-b8cc-e2fad1165450"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:45.383138 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.382999    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "dd491ac7-9aa5-4d43-b8cc-e2fad1165450" (UID: "dd491ac7-9aa5-4d43-b8cc-e2fad1165450"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:17:45.383138 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.383119    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "dd491ac7-9aa5-4d43-b8cc-e2fad1165450" (UID: "dd491ac7-9aa5-4d43-b8cc-e2fad1165450"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:45.383338 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.383317    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-data" (OuterVolumeSpecName: "istio-data") pod "dd491ac7-9aa5-4d43-b8cc-e2fad1165450" (UID: "dd491ac7-9aa5-4d43-b8cc-e2fad1165450"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:45.383535 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.383518    2589 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.383611 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.383540    2589 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.383611 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.383555    2589 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.383611 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.383569    2589 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.383611 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.383581    2589 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.384936 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.384895    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-token" (OuterVolumeSpecName: "istio-token") pod "dd491ac7-9aa5-4d43-b8cc-e2fad1165450" (UID: "dd491ac7-9aa5-4d43-b8cc-e2fad1165450"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:45.385317 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.385295    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-kube-api-access-jzhfk" (OuterVolumeSpecName: "kube-api-access-jzhfk") pod "dd491ac7-9aa5-4d43-b8cc-e2fad1165450" (UID: "dd491ac7-9aa5-4d43-b8cc-e2fad1165450"). InnerVolumeSpecName "kube-api-access-jzhfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:17:45.385570 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.385550    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "dd491ac7-9aa5-4d43-b8cc-e2fad1165450" (UID: "dd491ac7-9aa5-4d43-b8cc-e2fad1165450"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:17:45.385570 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.385538    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "dd491ac7-9aa5-4d43-b8cc-e2fad1165450" (UID: "dd491ac7-9aa5-4d43-b8cc-e2fad1165450"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:17:45.484502 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.484410    2589 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.484502 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.484439    2589 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.484502 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.484456    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhfk\" (UniqueName: \"kubernetes.io/projected/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-kube-api-access-jzhfk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.484502 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.484473    2589 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dd491ac7-9aa5-4d43-b8cc-e2fad1165450-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:17:45.619965 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.619935    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:45.687129 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.687097    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"]
Sep 17 18:17:45.691619 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.691590    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4390f8ca-gw-openshift-default-768c8f688d-bf9pv"]
Sep 17 18:17:45.701618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.701588    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:45.764091 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.764054    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"]
Sep 17 18:17:45.764411 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.764400    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy"
Sep 17 18:17:45.764456 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.764413    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy"
Sep 17 18:17:45.764492 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.764487    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy"
Sep 17 18:17:45.767691 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.767662    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:17:45.768415 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.768385    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:45.770087 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.770047    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7fd35bf2-dockercfg-5xg57\""
Sep 17 18:17:45.770215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.770047    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7fd35bf2\""
Sep 17 18:17:45.779251 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.779226    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"]
Sep 17 18:17:45.890268 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.890233    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrgsj\" (UniqueName: \"kubernetes.io/projected/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-kube-api-access-xrgsj\") pod \"payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv\" (UID: \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\") " pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:17:45.890268 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.890271    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv\" (UID: \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\") " pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:17:45.991470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.991423    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrgsj\" (UniqueName: \"kubernetes.io/projected/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-kube-api-access-xrgsj\") pod \"payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv\" (UID: \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\") " pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:17:45.991628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.991480    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv\" (UID: \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\") " pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:17:45.992218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.992198    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-plugins-config-volume\") pod \"payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv\" (UID: \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\") " pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:17:45.999107 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:45.999087    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrgsj\" (UniqueName: \"kubernetes.io/projected/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-kube-api-access-xrgsj\") pod \"payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv\" (UID: \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\") " pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:17:46.041487 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:46.041390    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:46.079387 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:46.079355    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:17:46.351404 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:46.351381    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"]
Sep 17 18:17:46.353915 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:17:46.353887    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf48d0410_73fd_4437_bcf6_8d25a0bce1c8.slice/crio-5e1cc321a63f8f8498bff45a21f5044abb39cb5364cc0b3c1a5af0fec5191561 WatchSource:0}: Error finding container 5e1cc321a63f8f8498bff45a21f5044abb39cb5364cc0b3c1a5af0fec5191561: Status 404 returned error can't find the container with id 5e1cc321a63f8f8498bff45a21f5044abb39cb5364cc0b3c1a5af0fec5191561
Sep 17 18:17:46.364820 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:46.364796    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv" event={"ID":"f48d0410-73fd-4437-bcf6-8d25a0bce1c8","Type":"ContainerStarted","Data":"5e1cc321a63f8f8498bff45a21f5044abb39cb5364cc0b3c1a5af0fec5191561"}
Sep 17 18:17:46.602897 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:46.602807    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:17:46.603616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:46.603585    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:17:46.683947 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:46.683920    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" path="/var/lib/kubelet/pods/dd491ac7-9aa5-4d43-b8cc-e2fad1165450/volumes"
Sep 17 18:17:46.744060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:46.744032    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:47.371162 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:47.371126    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv" event={"ID":"f48d0410-73fd-4437-bcf6-8d25a0bce1c8","Type":"ContainerStarted","Data":"2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307"}
Sep 17 18:17:47.371616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:47.371251    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:17:47.388171 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:47.388127    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv" podStartSLOduration=2.388115586 podStartE2EDuration="2.388115586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:17:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:17:47.386703813 +0000 UTC m=+903.066389666" watchObservedRunningTime="2026-09-17 18:17:47.388115586 +0000 UTC m=+903.067801440"
Sep 17 18:17:47.590500 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:47.590470    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:47.796581 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:47.796548    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:17:48.385030 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:48.384977    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:17:58.378456 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:17:58.378419    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:18:00.499534 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:00.499499    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:18:03.127336 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.127296    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:03.783233 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.783198    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"]
Sep 17 18:18:03.783572 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.783560    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy"
Sep 17 18:18:03.783617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.783575    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy"
Sep 17 18:18:03.783652 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.783646    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd491ac7-9aa5-4d43-b8cc-e2fad1165450" containerName="istio-proxy"
Sep 17 18:18:03.786837 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.786822    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.790251 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.790224    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"kube-root-ca.crt\""
Sep 17 18:18:03.790251 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.790245    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"openshift-service-ca.crt\""
Sep 17 18:18:03.790429 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.790332    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"default-dockercfg-t6hzs\""
Sep 17 18:18:03.790429 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.790387    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"simulator-w0-5d8053-kserve-self-signed-certs\""
Sep 17 18:18:03.795798 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.795776    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"]
Sep 17 18:18:03.838113 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.838088    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-home\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.838228 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.838117    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-model-cache\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.838228 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.838136    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-dshm\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.838228 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.838157    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7ce848-838c-4416-9e96-0c6ad8208f50-tls-certs\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.838394 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.838296    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-tmp-dir\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.838394 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.838371    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jrbr\" (UniqueName: \"kubernetes.io/projected/6c7ce848-838c-4416-9e96-0c6ad8208f50-kube-api-access-9jrbr\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.895407 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.895380    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:03.939651 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.939624    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-tmp-dir\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.939769 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.939682    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jrbr\" (UniqueName: \"kubernetes.io/projected/6c7ce848-838c-4416-9e96-0c6ad8208f50-kube-api-access-9jrbr\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.939769 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.939719    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-home\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.939769 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.939748    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-model-cache\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.939923 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.939775    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-dshm\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.939923 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.939807    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7ce848-838c-4416-9e96-0c6ad8208f50-tls-certs\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.940137 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.940117    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-tmp-dir\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.940189 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.940131    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-home\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.940241 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.940219    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-model-cache\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.942134 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.942116    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-dshm\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.942312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.942297    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7ce848-838c-4416-9e96-0c6ad8208f50-tls-certs\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:03.948944 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:03.948926    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jrbr\" (UniqueName: \"kubernetes.io/projected/6c7ce848-838c-4416-9e96-0c6ad8208f50-kube-api-access-9jrbr\") pod \"simulator-w0-5d8053-kserve-8458c65846-jsvm9\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") " pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:04.097444 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:04.097357    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:04.376486 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:04.376424    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"]
Sep 17 18:18:04.378808 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:18:04.378778    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c7ce848_838c_4416_9e96_0c6ad8208f50.slice/crio-091aa732480af87fb1ec5ee8aceedae76d0459d7be63a3813d7f6b24b1f7b04b WatchSource:0}: Error finding container 091aa732480af87fb1ec5ee8aceedae76d0459d7be63a3813d7f6b24b1f7b04b: Status 404 returned error can't find the container with id 091aa732480af87fb1ec5ee8aceedae76d0459d7be63a3813d7f6b24b1f7b04b
Sep 17 18:18:04.451424 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:04.451394    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9" event={"ID":"6c7ce848-838c-4416-9e96-0c6ad8208f50","Type":"ContainerStarted","Data":"22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8"}
Sep 17 18:18:04.451574 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:04.451437    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9" event={"ID":"6c7ce848-838c-4416-9e96-0c6ad8208f50","Type":"ContainerStarted","Data":"091aa732480af87fb1ec5ee8aceedae76d0459d7be63a3813d7f6b24b1f7b04b"}
Sep 17 18:18:06.048715 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:06.048673    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:07.433443 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:07.433405    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:08.348723 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:08.348686    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:09.188023 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:09.187971    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:09.862810 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:09.862775    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:10.723307 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:10.723265    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:11.435461 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:11.435428    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:12.228432 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:12.228395    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:12.697936 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:12.697880    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9" podStartSLOduration=9.697865523 podStartE2EDuration="9.697865523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:04.467180141 +0000 UTC m=+920.146865996" watchObservedRunningTime="2026-09-17 18:18:12.697865523 +0000 UTC m=+928.377551376"
Sep 17 18:18:12.698869 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:12.698851    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:18:13.034293 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:13.034263    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:13.774307 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:13.774274    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:14.098085 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.097978    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:14.098085 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.098056    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:14.110559 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.110534    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:14.398753 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.398666    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:14.429642 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.429606    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"]
Sep 17 18:18:14.429832 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.429811    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv" podUID="f48d0410-73fd-4437-bcf6-8d25a0bce1c8" containerName="payload-processing" containerID="cri-o://2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307" gracePeriod=30
Sep 17 18:18:14.462729 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.462701    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:14.512575 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.512551    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:18:14.533432 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.533402    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:14.595437 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.595404    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:14.680657 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.680633    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:18:14.683833 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.683774    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:14.731658 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.731624    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xrgsj\" (UniqueName: \"kubernetes.io/projected/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-kube-api-access-xrgsj\") pod \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\" (UID: \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\") "
Sep 17 18:18:14.731814 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.731719    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-plugins-config-volume\") pod \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\" (UID: \"f48d0410-73fd-4437-bcf6-8d25a0bce1c8\") "
Sep 17 18:18:14.732112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.732079    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f48d0410-73fd-4437-bcf6-8d25a0bce1c8" (UID: "f48d0410-73fd-4437-bcf6-8d25a0bce1c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:14.733924 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.733903    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-kube-api-access-xrgsj" (OuterVolumeSpecName: "kube-api-access-xrgsj") pod "f48d0410-73fd-4437-bcf6-8d25a0bce1c8" (UID: "f48d0410-73fd-4437-bcf6-8d25a0bce1c8"). InnerVolumeSpecName "kube-api-access-xrgsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:18:14.833632 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.833598    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:18:14.833632 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:14.833633    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrgsj\" (UniqueName: \"kubernetes.io/projected/f48d0410-73fd-4437-bcf6-8d25a0bce1c8-kube-api-access-xrgsj\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:18:15.028488 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.028455    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:15.194469 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.194429    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:15.508421 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.508392    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"
Sep 17 18:18:15.508421 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.508344    2589 generic.go:356] "Generic (PLEG): container finished" podID="f48d0410-73fd-4437-bcf6-8d25a0bce1c8" containerID="2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307" exitCode=0
Sep 17 18:18:15.508697 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.508443    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv" event={"ID":"f48d0410-73fd-4437-bcf6-8d25a0bce1c8","Type":"ContainerDied","Data":"2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307"}
Sep 17 18:18:15.508697 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.508506    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv" event={"ID":"f48d0410-73fd-4437-bcf6-8d25a0bce1c8","Type":"ContainerDied","Data":"5e1cc321a63f8f8498bff45a21f5044abb39cb5364cc0b3c1a5af0fec5191561"}
Sep 17 18:18:15.508697 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.508544    2589 scope.go:117] "RemoveContainer" containerID="2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307"
Sep 17 18:18:15.517489 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.517473    2589 scope.go:117] "RemoveContainer" containerID="2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307"
Sep 17 18:18:15.517773 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:18:15.517756    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307\": container with ID starting with 2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307 not found: ID does not exist" containerID="2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307"
Sep 17 18:18:15.517826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.517782    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307"} err="failed to get container status \"2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307\": rpc error: code = NotFound desc = could not find container \"2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307\": container with ID starting with 2424a352a2384a2601f14da3ffab79315bfea1e572211131cc78655e5de30307 not found: ID does not exist"
Sep 17 18:18:15.531921 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.531900    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"]
Sep 17 18:18:15.537055 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.536999    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7fd35bf2-8887d647c-zhrbv"]
Sep 17 18:18:15.767649 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.767556    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:15.845327 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:15.845274    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:16.173508 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:16.173421    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:16.497881 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:16.497799    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:16.683684 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:16.683646    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f48d0410-73fd-4437-bcf6-8d25a0bce1c8" path="/var/lib/kubelet/pods/f48d0410-73fd-4437-bcf6-8d25a0bce1c8/volumes"
Sep 17 18:18:16.749650 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:16.749574    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:16.928165 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:16.928131    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:17.757119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:17.757085    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:18.540183 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:18.540147    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:19.044567 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:19.044531    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:19.132835 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:19.132794    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:19.450533 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:19.450439    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:20.191898 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:20.191864    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:21.034322 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:21.034283    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:21.821843 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:21.821806    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:22.514245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:22.514218    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:23.365028 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:23.364978    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:23.466978 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:23.466941    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:23.565959 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:23.565928    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:23.878782 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:23.878746    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:23.981757 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:23.981721    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:35.303707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:35.303672    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:18:38.083899 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.083860    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"]
Sep 17 18:18:38.084291 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.084239    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f48d0410-73fd-4437-bcf6-8d25a0bce1c8" containerName="payload-processing"
Sep 17 18:18:38.084291 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.084251    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="f48d0410-73fd-4437-bcf6-8d25a0bce1c8" containerName="payload-processing"
Sep 17 18:18:38.084369 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.084309    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="f48d0410-73fd-4437-bcf6-8d25a0bce1c8" containerName="payload-processing"
Sep 17 18:18:38.087073 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.087056    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.089488 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.089466    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c31adf\"/\"openshift-service-ca.crt\""
Sep 17 18:18:38.090476 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.090449    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c31adf\"/\"default-dockercfg-kpcrb\""
Sep 17 18:18:38.090637 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.090496    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c31adf\"/\"kube-root-ca.crt\""
Sep 17 18:18:38.090637 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.090504    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c31adf\"/\"test-model-c31adf-kserve-self-signed-certs\""
Sep 17 18:18:38.096045 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.096022    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"]
Sep 17 18:18:38.233564 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.233530    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsrch\" (UniqueName: \"kubernetes.io/projected/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-kube-api-access-zsrch\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.233738 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.233615    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tls-certs\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.233738 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.233659    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-home\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.233738 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.233681    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tmp-dir\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.233738 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.233719    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-dshm\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.233913 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.233782    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-model-cache\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.334266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.334180    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-model-cache\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.334266 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.334232    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsrch\" (UniqueName: \"kubernetes.io/projected/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-kube-api-access-zsrch\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.334475 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.334290    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tls-certs\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.334475 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.334320    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-home\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.334475 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.334335    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tmp-dir\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.334475 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.334357    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-dshm\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.334675 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.334586    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-model-cache\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.334763 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.334705    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-home\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.334763 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.334716    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tmp-dir\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.337017 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.336969    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-dshm\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.337017 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.336982    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tls-certs\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.343416 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.343392    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsrch\" (UniqueName: \"kubernetes.io/projected/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-kube-api-access-zsrch\") pod \"test-model-c31adf-kserve-5b98bb7977-lqvnk\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") " pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.396688 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.396660    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:38.497210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.497179    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:38.670386 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:38.670362    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"]
Sep 17 18:18:38.672565 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:18:38.672536    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf32aa8f2_ecc8_43ef_8fb3_67e11edf43a4.slice/crio-710c7208d3eb666508ccd422b14a00a7696c1374017263d39912f8ed0578a2d2 WatchSource:0}: Error finding container 710c7208d3eb666508ccd422b14a00a7696c1374017263d39912f8ed0578a2d2: Status 404 returned error can't find the container with id 710c7208d3eb666508ccd422b14a00a7696c1374017263d39912f8ed0578a2d2
Sep 17 18:18:39.623773 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:39.623739    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk" event={"ID":"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4","Type":"ContainerStarted","Data":"ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f"}
Sep 17 18:18:39.623773 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:39.623781    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk" event={"ID":"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4","Type":"ContainerStarted","Data":"710c7208d3eb666508ccd422b14a00a7696c1374017263d39912f8ed0578a2d2"}
Sep 17 18:18:39.641948 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:39.641901    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk" podStartSLOduration=1.641885502 podStartE2EDuration="1.641885502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:39.640840457 +0000 UTC m=+955.320526313" watchObservedRunningTime="2026-09-17 18:18:39.641885502 +0000 UTC m=+955.321571356"
Sep 17 18:18:45.597627 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:45.597590    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:46.519439 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:46.519392    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:47.255568 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:47.255533    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:47.944642 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:47.944608    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:48.397427 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:48.397393    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:48.397851 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:48.397454    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:48.409744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:48.409719    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:48.649068 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:48.648982    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:48.676576 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:48.676551    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:18:49.198402 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.198366    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"]
Sep 17 18:18:49.202991 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.202975    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"
Sep 17 18:18:49.205250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.205233    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:18:49.205327 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.205233    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:18:49.209562 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.209478    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"]
Sep 17 18:18:49.226174 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.226144    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0432a945-0ebb-4509-a3e3-f7afaab2df09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b200d5-mtsq5\" (UID: \"0432a945-0ebb-4509-a3e3-f7afaab2df09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"
Sep 17 18:18:49.326867 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.326834    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0432a945-0ebb-4509-a3e3-f7afaab2df09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b200d5-mtsq5\" (UID: \"0432a945-0ebb-4509-a3e3-f7afaab2df09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"
Sep 17 18:18:49.327554 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.327532    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0432a945-0ebb-4509-a3e3-f7afaab2df09-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-b200d5-mtsq5\" (UID: \"0432a945-0ebb-4509-a3e3-f7afaab2df09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"
Sep 17 18:18:49.514086 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.514055    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"
Sep 17 18:18:49.752215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.752185    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:49.784401 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.784378    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"]
Sep 17 18:18:49.786479 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:18:49.786441    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0432a945_0ebb_4509_a3e3_f7afaab2df09.slice/crio-9928d2f3837e0205925a6e466eda2e3f4466e7b08dc80cd92c399be2d90893f7 WatchSource:0}: Error finding container 9928d2f3837e0205925a6e466eda2e3f4466e7b08dc80cd92c399be2d90893f7: Status 404 returned error can't find the container with id 9928d2f3837e0205925a6e466eda2e3f4466e7b08dc80cd92c399be2d90893f7
Sep 17 18:18:49.788584 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:49.788562    2589 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 17 18:18:50.467027 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:50.466958    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:50.676373 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:50.676338    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5" event={"ID":"0432a945-0ebb-4509-a3e3-f7afaab2df09","Type":"ContainerStarted","Data":"9928d2f3837e0205925a6e466eda2e3f4466e7b08dc80cd92c399be2d90893f7"}
Sep 17 18:18:51.398775 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:51.398722    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:52.107681 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:52.107573    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:52.686829 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:52.686744    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5" event={"ID":"0432a945-0ebb-4509-a3e3-f7afaab2df09","Type":"ContainerStarted","Data":"4b5a2479c45231fe78ef8485deb252a31d4e4ac0eb898f4a03b2c8668d56b371"}
Sep 17 18:18:52.703076 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:52.703024    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5" podStartSLOduration=1.081432553 podStartE2EDuration="3.702992941s" totalImagesPullingTime="2.621560388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:18:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:18:52.702174071 +0000 UTC m=+968.381859926" watchObservedRunningTime="2026-09-17 18:18:52.702992941 +0000 UTC m=+968.382678830"
Sep 17 18:18:52.928507 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:52.928470    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:53.641430 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:53.641393    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:53.693038 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:53.692982    2589 generic.go:356] "Generic (PLEG): container finished" podID="0432a945-0ebb-4509-a3e3-f7afaab2df09" containerID="4b5a2479c45231fe78ef8485deb252a31d4e4ac0eb898f4a03b2c8668d56b371" exitCode=0
Sep 17 18:18:53.693203 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:53.693069    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5" event={"ID":"0432a945-0ebb-4509-a3e3-f7afaab2df09","Type":"ContainerDied","Data":"4b5a2479c45231fe78ef8485deb252a31d4e4ac0eb898f4a03b2c8668d56b371"}
Sep 17 18:18:53.826088 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:53.826060    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:53.926443 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:53.926364    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:54.370868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:54.370833    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:54.832708 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:54.832687    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"
Sep 17 18:18:54.881067 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:54.881039    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0432a945-0ebb-4509-a3e3-f7afaab2df09-maas-api-service-ca\") pod \"0432a945-0ebb-4509-a3e3-f7afaab2df09\" (UID: \"0432a945-0ebb-4509-a3e3-f7afaab2df09\") "
Sep 17 18:18:54.881385 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:54.881366    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0432a945-0ebb-4509-a3e3-f7afaab2df09-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0432a945-0ebb-4509-a3e3-f7afaab2df09" (UID: "0432a945-0ebb-4509-a3e3-f7afaab2df09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:18:54.981690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:54.981621    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0432a945-0ebb-4509-a3e3-f7afaab2df09-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:18:55.703311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:55.703233    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5" event={"ID":"0432a945-0ebb-4509-a3e3-f7afaab2df09","Type":"ContainerDied","Data":"9928d2f3837e0205925a6e466eda2e3f4466e7b08dc80cd92c399be2d90893f7"}
Sep 17 18:18:55.703311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:55.703254    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"
Sep 17 18:18:55.703311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:55.703291    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9928d2f3837e0205925a6e466eda2e3f4466e7b08dc80cd92c399be2d90893f7"
Sep 17 18:18:58.926945 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:58.926908    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:18:59.854853 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:59.854818    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:18:59.885050 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:59.884997    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"]
Sep 17 18:18:59.885274 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:59.885245    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8" podUID="1d678436-9496-4e8f-b87e-6c5975966f44" containerName="payload-processing" containerID="cri-o://af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a" gracePeriod=30
Sep 17 18:18:59.893659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:59.893344    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"]
Sep 17 18:18:59.893659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:59.893570    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2" podUID="8f9bf969-372f-42f4-b7e5-f99acde0d955" containerName="payload-pre-processing" containerID="cri-o://0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02" gracePeriod=30
Sep 17 18:18:59.934262 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:18:59.934219    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:00.003462 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.003406    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:00.072056 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.071990    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"]
Sep 17 18:19:00.076973 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.076950    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-b200d5-mtsq5"]
Sep 17 18:19:00.081913 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.081876    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:00.139089 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.139063    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:19:00.161112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.161075    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:00.171364 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.171340    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:19:00.224069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.224037    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s59mq\" (UniqueName: \"kubernetes.io/projected/1d678436-9496-4e8f-b87e-6c5975966f44-kube-api-access-s59mq\") pod \"1d678436-9496-4e8f-b87e-6c5975966f44\" (UID: \"1d678436-9496-4e8f-b87e-6c5975966f44\") "
Sep 17 18:19:00.224252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.224098    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d678436-9496-4e8f-b87e-6c5975966f44-plugins-config-volume\") pod \"1d678436-9496-4e8f-b87e-6c5975966f44\" (UID: \"1d678436-9496-4e8f-b87e-6c5975966f44\") "
Sep 17 18:19:00.224252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.224159    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pb2bm\" (UniqueName: \"kubernetes.io/projected/8f9bf969-372f-42f4-b7e5-f99acde0d955-kube-api-access-pb2bm\") pod \"8f9bf969-372f-42f4-b7e5-f99acde0d955\" (UID: \"8f9bf969-372f-42f4-b7e5-f99acde0d955\") "
Sep 17 18:19:00.224252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.224195    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9bf969-372f-42f4-b7e5-f99acde0d955-plugins-config-volume\") pod \"8f9bf969-372f-42f4-b7e5-f99acde0d955\" (UID: \"8f9bf969-372f-42f4-b7e5-f99acde0d955\") "
Sep 17 18:19:00.224576 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.224551    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d678436-9496-4e8f-b87e-6c5975966f44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1d678436-9496-4e8f-b87e-6c5975966f44" (UID: "1d678436-9496-4e8f-b87e-6c5975966f44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:00.224648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.224611    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f9bf969-372f-42f4-b7e5-f99acde0d955-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8f9bf969-372f-42f4-b7e5-f99acde0d955" (UID: "8f9bf969-372f-42f4-b7e5-f99acde0d955"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:00.226483 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.226454    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d678436-9496-4e8f-b87e-6c5975966f44-kube-api-access-s59mq" (OuterVolumeSpecName: "kube-api-access-s59mq") pod "1d678436-9496-4e8f-b87e-6c5975966f44" (UID: "1d678436-9496-4e8f-b87e-6c5975966f44"). InnerVolumeSpecName "kube-api-access-s59mq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:00.226566 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.226488    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f9bf969-372f-42f4-b7e5-f99acde0d955-kube-api-access-pb2bm" (OuterVolumeSpecName: "kube-api-access-pb2bm") pod "8f9bf969-372f-42f4-b7e5-f99acde0d955" (UID: "8f9bf969-372f-42f4-b7e5-f99acde0d955"). InnerVolumeSpecName "kube-api-access-pb2bm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:00.325318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.325287    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s59mq\" (UniqueName: \"kubernetes.io/projected/1d678436-9496-4e8f-b87e-6c5975966f44-kube-api-access-s59mq\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:00.325318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.325313    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d678436-9496-4e8f-b87e-6c5975966f44-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:00.325318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.325324    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pb2bm\" (UniqueName: \"kubernetes.io/projected/8f9bf969-372f-42f4-b7e5-f99acde0d955-kube-api-access-pb2bm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:00.325557 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.325332    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f9bf969-372f-42f4-b7e5-f99acde0d955-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:00.414507 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.414423    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:00.547524 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.547483    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:00.683511 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.683435    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0432a945-0ebb-4509-a3e3-f7afaab2df09" path="/var/lib/kubelet/pods/0432a945-0ebb-4509-a3e3-f7afaab2df09/volumes"
Sep 17 18:19:00.727180 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.727146    2589 generic.go:356] "Generic (PLEG): container finished" podID="8f9bf969-372f-42f4-b7e5-f99acde0d955" containerID="0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02" exitCode=0
Sep 17 18:19:00.727328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.727202    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"
Sep 17 18:19:00.727328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.727239    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2" event={"ID":"8f9bf969-372f-42f4-b7e5-f99acde0d955","Type":"ContainerDied","Data":"0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02"}
Sep 17 18:19:00.727328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.727291    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2" event={"ID":"8f9bf969-372f-42f4-b7e5-f99acde0d955","Type":"ContainerDied","Data":"a3a5fb7d02ba8d7ea745cb317343bc3c73f3e3ed1042cb3af5d1c650b00cc934"}
Sep 17 18:19:00.727328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.727316    2589 scope.go:117] "RemoveContainer" containerID="0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02"
Sep 17 18:19:00.728597 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.728574    2589 generic.go:356] "Generic (PLEG): container finished" podID="1d678436-9496-4e8f-b87e-6c5975966f44" containerID="af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a" exitCode=0
Sep 17 18:19:00.728676 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.728625    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"
Sep 17 18:19:00.728718 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.728668    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8" event={"ID":"1d678436-9496-4e8f-b87e-6c5975966f44","Type":"ContainerDied","Data":"af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a"}
Sep 17 18:19:00.728772 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.728728    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8" event={"ID":"1d678436-9496-4e8f-b87e-6c5975966f44","Type":"ContainerDied","Data":"e33d49a9a68a60d391e0f82f3050d0c510b8c6aed6330484c0cb14768d9062c2"}
Sep 17 18:19:00.736332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.736316    2589 scope.go:117] "RemoveContainer" containerID="0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02"
Sep 17 18:19:00.736581 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:19:00.736563    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02\": container with ID starting with 0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02 not found: ID does not exist" containerID="0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02"
Sep 17 18:19:00.736629 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.736588    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02"} err="failed to get container status \"0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02\": rpc error: code = NotFound desc = could not find container \"0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02\": container with ID starting with 0b6539d248b0e02a251fff7b1e964c5efd8ffe914fb9b766eaa9b82223050d02 not found: ID does not exist"
Sep 17 18:19:00.736629 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.736604    2589 scope.go:117] "RemoveContainer" containerID="af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a"
Sep 17 18:19:00.745093 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.745067    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"]
Sep 17 18:19:00.747282 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.747266    2589 scope.go:117] "RemoveContainer" containerID="af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a"
Sep 17 18:19:00.747545 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:19:00.747526    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a\": container with ID starting with af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a not found: ID does not exist" containerID="af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a"
Sep 17 18:19:00.747604 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.747554    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a"} err="failed to get container status \"af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a\": rpc error: code = NotFound desc = could not find container \"af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a\": container with ID starting with af0c26de00cbf57b9e0d8d1368a177f110f605123fe0796ce4bdb9b38855ad3a not found: ID does not exist"
Sep 17 18:19:00.751219 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.751196    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-b200d5-65cd94894b-vbcx2"]
Sep 17 18:19:00.761134 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.761112    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"]
Sep 17 18:19:00.764579 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:00.764559    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-b200d5-d84d9c6f9-n55r8"]
Sep 17 18:19:01.242033 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:01.241956    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:02.165689 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:02.165649    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:02.683017 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:02.682968    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d678436-9496-4e8f-b87e-6c5975966f44" path="/var/lib/kubelet/pods/1d678436-9496-4e8f-b87e-6c5975966f44/volumes"
Sep 17 18:19:02.683364 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:02.683350    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f9bf969-372f-42f4-b7e5-f99acde0d955" path="/var/lib/kubelet/pods/8f9bf969-372f-42f4-b7e5-f99acde0d955/volumes"
Sep 17 18:19:02.921433 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:02.921404    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:03.798480 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:03.798435    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:04.422359 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:04.422329    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:04.489961 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:04.489926    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:04.589827 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:04.589794    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:04.648229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:04.648182    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:04.788187 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:04.788148    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:05.150185 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:05.150102    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:05.502215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:05.502121    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:05.600801 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:05.600765    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:06.062721 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:06.062689    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:06.942259 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:06.942225    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:07.052951 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:07.052906    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:07.827800 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:07.827771    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:08.655981 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:08.655948    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:13.020897 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:13.020859    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:13.126643 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:13.126606    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:13.218563 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:13.218530    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:13.319480 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:13.319389    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:13.698384 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:13.698302    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:15.201918 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:15.201878    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:19:24.016357 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:24.016323    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:39.582856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:39.582819    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:41.912783 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:41.912745    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:42.178365 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:42.178274    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"]
Sep 17 18:19:42.178549 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:42.178521    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy" containerID="cri-o://3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b" gracePeriod=30
Sep 17 18:19:42.240536 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:42.240507    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:42.626861 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:42.626821    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:42.917934 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:42.917837    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:43.175390 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:43.175314    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:43.280122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:43.280082    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:45.414047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:45.413998    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:46.094508 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:46.094463    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:46.197368 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:46.197336    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:46.306519 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:46.306487    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:46.412578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:46.412494    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:47.272296 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.272267    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:47.419700 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.419676    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:19:47.525418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525392    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-envoy\") pod \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") "
Sep 17 18:19:47.525555 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525432    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-podinfo\") pod \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") "
Sep 17 18:19:47.525555 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525483    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-token\") pod \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") "
Sep 17 18:19:47.525555 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525504    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-certs\") pod \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") "
Sep 17 18:19:47.525555 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525525    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istiod-ca-cert\") pod \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") "
Sep 17 18:19:47.525776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525567    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-credential-socket\") pod \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") "
Sep 17 18:19:47.525776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525603    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lgsv\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-kube-api-access-4lgsv\") pod \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") "
Sep 17 18:19:47.525776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525638    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-data\") pod \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") "
Sep 17 18:19:47.525776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525681    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-socket\") pod \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\" (UID: \"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca\") "
Sep 17 18:19:47.525985 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525817    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" (UID: "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:19:47.525985 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.525918    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" (UID: "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:19:47.526130 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.526077    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" (UID: "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:19:47.526196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.526175    2589 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:47.526252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.526201    2589 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:47.526252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.526215    2589 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:47.526352 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.526324    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" (UID: "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:19:47.526390 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.526356    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-data" (OuterVolumeSpecName: "istio-data") pod "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" (UID: "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:19:47.527903 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.527877    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" (UID: "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:19:47.527903 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.527879    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" (UID: "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:19:47.528281 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.528254    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-token" (OuterVolumeSpecName: "istio-token") pod "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" (UID: "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:47.528371 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.528291    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-kube-api-access-4lgsv" (OuterVolumeSpecName: "kube-api-access-4lgsv") pod "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" (UID: "1ef3f16c-6cff-4b3c-bb2c-7f61196288ca"). InnerVolumeSpecName "kube-api-access-4lgsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:19:47.627020 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.626982    2589 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:47.627149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.627026    2589 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:47.627149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.627035    2589 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:47.627149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.627043    2589 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:47.627149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.627051    2589 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:47.627149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.627059    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lgsv\" (UniqueName: \"kubernetes.io/projected/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca-kube-api-access-4lgsv\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:19:47.949633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.949544    2589 generic.go:356] "Generic (PLEG): container finished" podID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerID="3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b" exitCode=0
Sep 17 18:19:47.949633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.949586    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" event={"ID":"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca","Type":"ContainerDied","Data":"3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b"}
Sep 17 18:19:47.949633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.949597    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"
Sep 17 18:19:47.949633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.949622    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v" event={"ID":"1ef3f16c-6cff-4b3c-bb2c-7f61196288ca","Type":"ContainerDied","Data":"5f097bdfaf50ed147ac30a473c8fc02df157ced48054fb97626cdbdb5f7893b2"}
Sep 17 18:19:47.949912 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.949644    2589 scope.go:117] "RemoveContainer" containerID="3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b"
Sep 17 18:19:47.961713 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.961694    2589 scope.go:117] "RemoveContainer" containerID="cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61"
Sep 17 18:19:47.969783 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.969763    2589 scope.go:117] "RemoveContainer" containerID="3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b"
Sep 17 18:19:47.970060 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:19:47.970038    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b\": container with ID starting with 3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b not found: ID does not exist" containerID="3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b"
Sep 17 18:19:47.970114 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.970069    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b"} err="failed to get container status \"3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b\": rpc error: code = NotFound desc = could not find container \"3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b\": container with ID starting with 3b0af6ef011ddeea26803dadc04ee65cd7f1bb9c53950aa54892dd90a0f0d71b not found: ID does not exist"
Sep 17 18:19:47.970114 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.970095    2589 scope.go:117] "RemoveContainer" containerID="cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61"
Sep 17 18:19:47.970350 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:19:47.970330    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61\": container with ID starting with cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61 not found: ID does not exist" containerID="cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61"
Sep 17 18:19:47.970410 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.970359    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61"} err="failed to get container status \"cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61\": rpc error: code = NotFound desc = could not find container \"cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61\": container with ID starting with cc4ab206deadd1ea3a35fc95dd64d3cf1f9b80fce5e406676cfd5782fc3d0f61 not found: ID does not exist"
Sep 17 18:19:47.979309 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.979277    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"]
Sep 17 18:19:47.990217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:47.990198    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-b200d5-openshift-default-595dcb7776-j5c7v"]
Sep 17 18:19:48.148102 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:48.148042    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:48.684221 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:48.684186    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" path="/var/lib/kubelet/pods/1ef3f16c-6cff-4b3c-bb2c-7f61196288ca/volumes"
Sep 17 18:19:49.086597 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:49.086542    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:49.921872 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:49.921836    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:19:50.076157 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.076023    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:50.178288 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.178161    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:50.648275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.648242    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:50.744504 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.744476    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:50.764592 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764563    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"]
Sep 17 18:19:50.764910 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764897    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d678436-9496-4e8f-b87e-6c5975966f44" containerName="payload-processing"
Sep 17 18:19:50.764953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764912    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d678436-9496-4e8f-b87e-6c5975966f44" containerName="payload-processing"
Sep 17 18:19:50.764953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764922    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy"
Sep 17 18:19:50.764953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764931    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy"
Sep 17 18:19:50.764953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764942    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy"
Sep 17 18:19:50.764953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764947    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy"
Sep 17 18:19:50.765135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764956    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0432a945-0ebb-4509-a3e3-f7afaab2df09" containerName="revoke-keys"
Sep 17 18:19:50.765135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764961    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="0432a945-0ebb-4509-a3e3-f7afaab2df09" containerName="revoke-keys"
Sep 17 18:19:50.765135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764969    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f9bf969-372f-42f4-b7e5-f99acde0d955" containerName="payload-pre-processing"
Sep 17 18:19:50.765135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.764974    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f9bf969-372f-42f4-b7e5-f99acde0d955" containerName="payload-pre-processing"
Sep 17 18:19:50.765135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.765051    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="0432a945-0ebb-4509-a3e3-f7afaab2df09" containerName="revoke-keys"
Sep 17 18:19:50.765135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.765059    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy"
Sep 17 18:19:50.765135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.765066    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ef3f16c-6cff-4b3c-bb2c-7f61196288ca" containerName="istio-proxy"
Sep 17 18:19:50.765135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.765072    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d678436-9496-4e8f-b87e-6c5975966f44" containerName="payload-processing"
Sep 17 18:19:50.765135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.765079    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f9bf969-372f-42f4-b7e5-f99acde0d955" containerName="payload-pre-processing"
Sep 17 18:19:50.768115 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.768096    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:19:50.770640 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.770623    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-83507f\""
Sep 17 18:19:50.780564 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.780539    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"]
Sep 17 18:19:50.829941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.829915    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:19:50.856301 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.856276    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a20355-1d56-40c2-b716-b4953754a746-plugins-config-volume\") pod \"payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x\" (UID: \"b6a20355-1d56-40c2-b716-b4953754a746\") " pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:19:50.856427 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.856317    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dmzrl\" (UniqueName: \"kubernetes.io/projected/b6a20355-1d56-40c2-b716-b4953754a746-kube-api-access-dmzrl\") pod \"payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x\" (UID: \"b6a20355-1d56-40c2-b716-b4953754a746\") " pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:19:50.957164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.957083    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a20355-1d56-40c2-b716-b4953754a746-plugins-config-volume\") pod \"payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x\" (UID: \"b6a20355-1d56-40c2-b716-b4953754a746\") " pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:19:50.957164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.957119    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dmzrl\" (UniqueName: \"kubernetes.io/projected/b6a20355-1d56-40c2-b716-b4953754a746-kube-api-access-dmzrl\") pod \"payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x\" (UID: \"b6a20355-1d56-40c2-b716-b4953754a746\") " pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:19:50.957757 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.957738    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a20355-1d56-40c2-b716-b4953754a746-plugins-config-volume\") pod \"payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x\" (UID: \"b6a20355-1d56-40c2-b716-b4953754a746\") " pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:19:50.965324 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:50.965298    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dmzrl\" (UniqueName: \"kubernetes.io/projected/b6a20355-1d56-40c2-b716-b4953754a746-kube-api-access-dmzrl\") pod \"payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x\" (UID: \"b6a20355-1d56-40c2-b716-b4953754a746\") " pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:19:51.078278 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:51.078245    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:19:51.348605 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:51.348510    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"]
Sep 17 18:19:51.350945 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:19:51.350909    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a20355_1d56_40c2_b716_b4953754a746.slice/crio-9f0d23b19607d62edddc691fcccbb967faec836aa0785c4b6dec354344a4a16b WatchSource:0}: Error finding container 9f0d23b19607d62edddc691fcccbb967faec836aa0785c4b6dec354344a4a16b: Status 404 returned error can't find the container with id 9f0d23b19607d62edddc691fcccbb967faec836aa0785c4b6dec354344a4a16b
Sep 17 18:19:51.971395 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:51.971358    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x" event={"ID":"b6a20355-1d56-40c2-b716-b4953754a746","Type":"ContainerStarted","Data":"ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d"}
Sep 17 18:19:51.971876 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:51.971418    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x" event={"ID":"b6a20355-1d56-40c2-b716-b4953754a746","Type":"ContainerStarted","Data":"9f0d23b19607d62edddc691fcccbb967faec836aa0785c4b6dec354344a4a16b"}
Sep 17 18:19:51.971876 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:51.971482    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:19:51.994968 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:19:51.994916    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x" podStartSLOduration=1.994899798 podStartE2EDuration="1.994899798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:19:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:19:51.99370344 +0000 UTC m=+1027.673389293" watchObservedRunningTime="2026-09-17 18:19:51.994899798 +0000 UTC m=+1027.674585652"
Sep 17 18:20:02.273799 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:02.273752    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:02.977814 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:02.977779    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:20:03.512831 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.512801    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"]
Sep 17 18:20:03.515153 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.515134    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.517462 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.517439    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"premium-w0-5d8053-kserve-self-signed-certs\""
Sep 17 18:20:03.525622 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.525599    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"]
Sep 17 18:20:03.544985 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.544952    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:03.569054 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.568991    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-dshm\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.569197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.569088    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-home\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.569197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.569110    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tmp-dir\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.569311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.569239    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-model-cache\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.569311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.569276    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhfct\" (UniqueName: \"kubernetes.io/projected/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-kube-api-access-vhfct\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.569392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.569325    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tls-certs\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.670448 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.670411    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-model-cache\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.670448 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.670445    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhfct\" (UniqueName: \"kubernetes.io/projected/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-kube-api-access-vhfct\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.670707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.670479    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tls-certs\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.670707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.670528    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-dshm\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.670707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.670559    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-home\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.670707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.670588    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tmp-dir\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.670917 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.670885    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-model-cache\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.670917 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.670906    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-home\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.671062 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.671043    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tmp-dir\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.673137 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.673109    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-dshm\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.673245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.673234    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tls-certs\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.679597 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.679576    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhfct\" (UniqueName: \"kubernetes.io/projected/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-kube-api-access-vhfct\") pod \"premium-w0-5d8053-kserve-69bfd4d48c-mm6qb\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") " pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:03.824733 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:03.824641    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:04.098170 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:04.098143    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"]
Sep 17 18:20:04.100552 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:20:04.100521    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod694cc9d0_dafe_4833_a7b8_e70e1eb7a662.slice/crio-e266825221d3dd7c184152fb2c7df8641cb2bc59a998a31574ad8ca4cd0375e7 WatchSource:0}: Error finding container e266825221d3dd7c184152fb2c7df8641cb2bc59a998a31574ad8ca4cd0375e7: Status 404 returned error can't find the container with id e266825221d3dd7c184152fb2c7df8641cb2bc59a998a31574ad8ca4cd0375e7
Sep 17 18:20:05.035831 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:05.035793    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb" event={"ID":"694cc9d0-dafe-4833-a7b8-e70e1eb7a662","Type":"ContainerStarted","Data":"5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85"}
Sep 17 18:20:05.036231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:05.035848    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb" event={"ID":"694cc9d0-dafe-4833-a7b8-e70e1eb7a662","Type":"ContainerStarted","Data":"e266825221d3dd7c184152fb2c7df8641cb2bc59a998a31574ad8ca4cd0375e7"}
Sep 17 18:20:05.053583 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:05.053539    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb" podStartSLOduration=2.053525042 podStartE2EDuration="2.053525042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:05.052477188 +0000 UTC m=+1040.732163054" watchObservedRunningTime="2026-09-17 18:20:05.053525042 +0000 UTC m=+1040.733210911"
Sep 17 18:20:08.477742 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:08.477698    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:08.603938 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:08.603903    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:08.728578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:08.728485    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:11.508625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:11.508588    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:20:12.436535 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:12.436498    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:13.152208 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:13.152177    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:13.825030 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:13.824970    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:13.825210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:13.825044    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:13.837759 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:13.837733    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:13.872677 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:13.872653    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:14.088391 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:14.088309    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:20:14.841053 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:14.841024    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:15.542900 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:15.542866    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:16.108953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:16.108917    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:16.330342 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:16.330302    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:19.161248 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:19.161218    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:19.908989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:19.908956    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:20.604218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:20.604182    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:25.936135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:25.936046    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:26.050826 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:26.050790    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:26.138133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:26.138092    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:26.237690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:26.237611    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:26.859069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:26.859043    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:26.948851 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:26.948823    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:27.052733 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:27.052696    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:28.497671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:28.497645    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:30.911679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:30.911642    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:20:34.746429 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:34.746391    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:34.811116 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:34.811081    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:35.396311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:35.396238    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:38.898481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.898436    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"]
Sep 17 18:20:38.901377 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.901361    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:38.904811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.904787    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-83507f\"/\"default-dockercfg-x7jkl\""
Sep 17 18:20:38.905045 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.904801    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-83507f\"/\"openshift-service-ca.crt\""
Sep 17 18:20:38.905045 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.904830    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-83507f\"/\"kube-root-ca.crt\""
Sep 17 18:20:38.905223 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.904855    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-83507f\"/\"explicit-model-83507f-kserve-self-signed-certs\""
Sep 17 18:20:38.911394 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.911370    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"]
Sep 17 18:20:38.922245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.922221    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:38.976507 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.976479    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-home\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:38.976628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.976533    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-model-cache\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:38.976628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.976572    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76267c15-7d53-4873-8328-a2ad0abd80d9-tls-certs\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:38.976707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.976634    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpghp\" (UniqueName: \"kubernetes.io/projected/76267c15-7d53-4873-8328-a2ad0abd80d9-kube-api-access-hpghp\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:38.976707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.976659    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-dshm\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:38.976707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:38.976695    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-tmp-dir\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.077071 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.077039    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-home\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.077231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.077108    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-model-cache\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.077231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.077137    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76267c15-7d53-4873-8328-a2ad0abd80d9-tls-certs\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.077359 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.077270    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpghp\" (UniqueName: \"kubernetes.io/projected/76267c15-7d53-4873-8328-a2ad0abd80d9-kube-api-access-hpghp\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.077359 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.077308    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-dshm\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.077359 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.077349    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-tmp-dir\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.077521 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.077486    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-home\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.077574 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.077529    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-model-cache\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.077734 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.077701    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-tmp-dir\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.079657 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.079634    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-dshm\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.079870 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.079851    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76267c15-7d53-4873-8328-a2ad0abd80d9-tls-certs\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.086165 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.086141    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpghp\" (UniqueName: \"kubernetes.io/projected/76267c15-7d53-4873-8328-a2ad0abd80d9-kube-api-access-hpghp\") pod \"explicit-model-83507f-kserve-59599f987f-7642z\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") " pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.212983 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.212911    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:39.483278 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:39.483252    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"]
Sep 17 18:20:39.485552 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:20:39.485523    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76267c15_7d53_4873_8328_a2ad0abd80d9.slice/crio-69830ce6c6e105f101e6502b830a576942e6ba798bf346d7f16868b02d5baadf WatchSource:0}: Error finding container 69830ce6c6e105f101e6502b830a576942e6ba798bf346d7f16868b02d5baadf: Status 404 returned error can't find the container with id 69830ce6c6e105f101e6502b830a576942e6ba798bf346d7f16868b02d5baadf
Sep 17 18:20:40.204224 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:40.204186    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z" event={"ID":"76267c15-7d53-4873-8328-a2ad0abd80d9","Type":"ContainerStarted","Data":"657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb"}
Sep 17 18:20:40.204613 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:40.204234    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z" event={"ID":"76267c15-7d53-4873-8328-a2ad0abd80d9","Type":"ContainerStarted","Data":"69830ce6c6e105f101e6502b830a576942e6ba798bf346d7f16868b02d5baadf"}
Sep 17 18:20:40.221581 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:40.221536    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z" podStartSLOduration=2.221522491 podStartE2EDuration="2.221522491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:40.220968293 +0000 UTC m=+1075.900654146" watchObservedRunningTime="2026-09-17 18:20:40.221522491 +0000 UTC m=+1075.901208338"
Sep 17 18:20:41.338840 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:41.338793    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:41.405411 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:41.405379    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:41.563152 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:41.563114    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:41.859849 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:41.859815    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:42.129929 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:42.129833    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:47.235955 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:47.235912    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:47.941413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:47.941381    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:48.037458 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:48.037426    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:49.037990 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:49.037955    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:49.213499 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:49.213460    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:49.213657 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:49.213519    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:49.225775 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:49.225749    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:49.257900 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:49.257869    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:20:49.928779 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:49.928747    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:50.701920 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:50.701887    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:51.544914 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:51.544877    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:52.341058 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.341000    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:52.438452 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.438417    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:52.592950 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.592863    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:52.836641 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.836611    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:52.920159 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.920077    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:20:52.976212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.976175    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"]
Sep 17 18:20:52.980654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.980636    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:20:52.983092 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.983070    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d62f3adc-dockercfg-hmxsh\""
Sep 17 18:20:52.983211 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.983076    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d62f3adc\""
Sep 17 18:20:52.989082 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:52.989054    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"]
Sep 17 18:20:53.097409 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:53.097377    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd1ea9-1f32-443f-9114-a5b1342d5498-plugins-config-volume\") pod \"payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh\" (UID: \"18fd1ea9-1f32-443f-9114-a5b1342d5498\") " pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:20:53.097562 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:53.097486    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4498\" (UniqueName: \"kubernetes.io/projected/18fd1ea9-1f32-443f-9114-a5b1342d5498-kube-api-access-g4498\") pod \"payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh\" (UID: \"18fd1ea9-1f32-443f-9114-a5b1342d5498\") " pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:20:53.198268 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:53.198178    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd1ea9-1f32-443f-9114-a5b1342d5498-plugins-config-volume\") pod \"payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh\" (UID: \"18fd1ea9-1f32-443f-9114-a5b1342d5498\") " pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:20:53.198423 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:53.198293    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4498\" (UniqueName: \"kubernetes.io/projected/18fd1ea9-1f32-443f-9114-a5b1342d5498-kube-api-access-g4498\") pod \"payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh\" (UID: \"18fd1ea9-1f32-443f-9114-a5b1342d5498\") " pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:20:53.198868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:53.198848    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd1ea9-1f32-443f-9114-a5b1342d5498-plugins-config-volume\") pod \"payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh\" (UID: \"18fd1ea9-1f32-443f-9114-a5b1342d5498\") " pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:20:53.206164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:53.206145    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4498\" (UniqueName: \"kubernetes.io/projected/18fd1ea9-1f32-443f-9114-a5b1342d5498-kube-api-access-g4498\") pod \"payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh\" (UID: \"18fd1ea9-1f32-443f-9114-a5b1342d5498\") " pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:20:53.292029 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:53.291973    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:20:53.564716 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:53.564646    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"]
Sep 17 18:20:53.567122 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:20:53.567090    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18fd1ea9_1f32_443f_9114_a5b1342d5498.slice/crio-1205068f8fc25a35c523a1930e3010f9c0c1a24492dcf160cb161ce45d9e33c0 WatchSource:0}: Error finding container 1205068f8fc25a35c523a1930e3010f9c0c1a24492dcf160cb161ce45d9e33c0: Status 404 returned error can't find the container with id 1205068f8fc25a35c523a1930e3010f9c0c1a24492dcf160cb161ce45d9e33c0
Sep 17 18:20:54.271800 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:54.271760    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh" event={"ID":"18fd1ea9-1f32-443f-9114-a5b1342d5498","Type":"ContainerStarted","Data":"cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f"}
Sep 17 18:20:54.271995 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:54.271811    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh" event={"ID":"18fd1ea9-1f32-443f-9114-a5b1342d5498","Type":"ContainerStarted","Data":"1205068f8fc25a35c523a1930e3010f9c0c1a24492dcf160cb161ce45d9e33c0"}
Sep 17 18:20:54.271995 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:54.271851    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:20:54.292311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:20:54.292269    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh" podStartSLOduration=2.292255125 podStartE2EDuration="2.292255125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:20:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:20:54.290431639 +0000 UTC m=+1089.970117493" watchObservedRunningTime="2026-09-17 18:20:54.292255125 +0000 UTC m=+1089.971941008"
Sep 17 18:21:05.278648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:05.278613    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:21:08.415958 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:08.415920    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:21:11.975108 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:11.975073    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"]
Sep 17 18:21:12.004864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:12.004836    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:12.391936 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:12.391899    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:13.227493 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:13.227451    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:14.020840 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:14.020811    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:14.738730 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:14.738683    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:15.539040 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:15.538959    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:16.227656 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:16.227621    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:17.130534 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:17.130492    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:17.840166 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:17.840133    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:18.022416 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:18.022382    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:18.715695 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:18.715657    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:18.806240 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:18.806210    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:19.106444 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:19.106411    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:19.533051 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:19.532999    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:20.353923 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:20.353891    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:21.106925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:21.106894    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:22.075528 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:22.075493    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:22.798745 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:22.798709    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:23.391026 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.390976    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:23.423479 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.423450    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"]
Sep 17 18:21:23.423818 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.423777    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x" podUID="b6a20355-1d56-40c2-b716-b4953754a746" containerName="payload-processing" containerID="cri-o://ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d" gracePeriod=30
Sep 17 18:21:23.491158 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.491122    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:23.590969 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.590933    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:23.672932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.672907    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:21:23.693101 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.693071    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:23.770743 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.770707    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dmzrl\" (UniqueName: \"kubernetes.io/projected/b6a20355-1d56-40c2-b716-b4953754a746-kube-api-access-dmzrl\") pod \"b6a20355-1d56-40c2-b716-b4953754a746\" (UID: \"b6a20355-1d56-40c2-b716-b4953754a746\") "
Sep 17 18:21:23.770901 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.770796    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a20355-1d56-40c2-b716-b4953754a746-plugins-config-volume\") pod \"b6a20355-1d56-40c2-b716-b4953754a746\" (UID: \"b6a20355-1d56-40c2-b716-b4953754a746\") "
Sep 17 18:21:23.771298 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.771268    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a20355-1d56-40c2-b716-b4953754a746-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b6a20355-1d56-40c2-b716-b4953754a746" (UID: "b6a20355-1d56-40c2-b716-b4953754a746"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:23.772946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.772919    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a20355-1d56-40c2-b716-b4953754a746-kube-api-access-dmzrl" (OuterVolumeSpecName: "kube-api-access-dmzrl") pod "b6a20355-1d56-40c2-b716-b4953754a746" (UID: "b6a20355-1d56-40c2-b716-b4953754a746"). InnerVolumeSpecName "kube-api-access-dmzrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:23.872479 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.872446    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dmzrl\" (UniqueName: \"kubernetes.io/projected/b6a20355-1d56-40c2-b716-b4953754a746-kube-api-access-dmzrl\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:23.872479 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.872475    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6a20355-1d56-40c2-b716-b4953754a746-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:23.962500 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:23.962406    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:24.063387 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.063348    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:24.412643 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.412609    2589 generic.go:356] "Generic (PLEG): container finished" podID="b6a20355-1d56-40c2-b716-b4953754a746" containerID="ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d" exitCode=0
Sep 17 18:21:24.413164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.412661    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"
Sep 17 18:21:24.413164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.412705    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x" event={"ID":"b6a20355-1d56-40c2-b716-b4953754a746","Type":"ContainerDied","Data":"ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d"}
Sep 17 18:21:24.413164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.412764    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x" event={"ID":"b6a20355-1d56-40c2-b716-b4953754a746","Type":"ContainerDied","Data":"9f0d23b19607d62edddc691fcccbb967faec836aa0785c4b6dec354344a4a16b"}
Sep 17 18:21:24.413164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.412791    2589 scope.go:117] "RemoveContainer" containerID="ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d"
Sep 17 18:21:24.422103 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.422083    2589 scope.go:117] "RemoveContainer" containerID="ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d"
Sep 17 18:21:24.422371 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:21:24.422351    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d\": container with ID starting with ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d not found: ID does not exist" containerID="ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d"
Sep 17 18:21:24.422441 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.422379    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d"} err="failed to get container status \"ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d\": rpc error: code = NotFound desc = could not find container \"ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d\": container with ID starting with ba6a8f583be574a24d02310e893b8fd6fc9c41781bb902cc24fe58a8e5a70d2d not found: ID does not exist"
Sep 17 18:21:24.433572 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.433552    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"]
Sep 17 18:21:24.439922 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.439901    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-83507f-6987f5558f-ncp7x"]
Sep 17 18:21:24.683230 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.683155    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6a20355-1d56-40c2-b716-b4953754a746" path="/var/lib/kubelet/pods/b6a20355-1d56-40c2-b716-b4953754a746/volumes"
Sep 17 18:21:24.947031 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:24.946922    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:25.694335 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:25.694296    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:25.797734 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:25.797695    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:25.919990 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:25.919952    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:26.187560 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.187530    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"]
Sep 17 18:21:26.187891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.187879    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a20355-1d56-40c2-b716-b4953754a746" containerName="payload-processing"
Sep 17 18:21:26.187939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.187893    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a20355-1d56-40c2-b716-b4953754a746" containerName="payload-processing"
Sep 17 18:21:26.187975 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.187953    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a20355-1d56-40c2-b716-b4953754a746" containerName="payload-processing"
Sep 17 18:21:26.192402 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.192382    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"
Sep 17 18:21:26.198212 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.198189    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:21:26.198340 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.198279    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:21:26.202483 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.202452    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"]
Sep 17 18:21:26.292571 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.292539    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/217776f6-a20d-4a8e-a86e-1a432a23e689-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh\" (UID: \"217776f6-a20d-4a8e-a86e-1a432a23e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"
Sep 17 18:21:26.393379 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.393339    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/217776f6-a20d-4a8e-a86e-1a432a23e689-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh\" (UID: \"217776f6-a20d-4a8e-a86e-1a432a23e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"
Sep 17 18:21:26.393953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.393932    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/217776f6-a20d-4a8e-a86e-1a432a23e689-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh\" (UID: \"217776f6-a20d-4a8e-a86e-1a432a23e689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"
Sep 17 18:21:26.504476 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.504452    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"
Sep 17 18:21:26.774428 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.774379    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"]
Sep 17 18:21:26.776739 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:21:26.776708    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod217776f6_a20d_4a8e_a86e_1a432a23e689.slice/crio-789e86bad693bd1c9a149f25694519f02676cbbedb72cf0bb24d2d6ffbacfb29 WatchSource:0}: Error finding container 789e86bad693bd1c9a149f25694519f02676cbbedb72cf0bb24d2d6ffbacfb29: Status 404 returned error can't find the container with id 789e86bad693bd1c9a149f25694519f02676cbbedb72cf0bb24d2d6ffbacfb29
Sep 17 18:21:26.927578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.927544    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:26.994333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:26.994297    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z" podUID="76267c15-7d53-4873-8328-a2ad0abd80d9" containerName="main" containerID="cri-o://657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb" gracePeriod=45
Sep 17 18:21:27.224675 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.224647    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:21:27.299772 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.299730    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-home\") pod \"76267c15-7d53-4873-8328-a2ad0abd80d9\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") "
Sep 17 18:21:27.299772 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.299774    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-model-cache\") pod \"76267c15-7d53-4873-8328-a2ad0abd80d9\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") "
Sep 17 18:21:27.300049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.299804    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-dshm\") pod \"76267c15-7d53-4873-8328-a2ad0abd80d9\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") "
Sep 17 18:21:27.300049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.299824    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-tmp-dir\") pod \"76267c15-7d53-4873-8328-a2ad0abd80d9\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") "
Sep 17 18:21:27.300049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.299854    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76267c15-7d53-4873-8328-a2ad0abd80d9-tls-certs\") pod \"76267c15-7d53-4873-8328-a2ad0abd80d9\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") "
Sep 17 18:21:27.300049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.299882    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpghp\" (UniqueName: \"kubernetes.io/projected/76267c15-7d53-4873-8328-a2ad0abd80d9-kube-api-access-hpghp\") pod \"76267c15-7d53-4873-8328-a2ad0abd80d9\" (UID: \"76267c15-7d53-4873-8328-a2ad0abd80d9\") "
Sep 17 18:21:27.300231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.300053    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-home" (OuterVolumeSpecName: "home") pod "76267c15-7d53-4873-8328-a2ad0abd80d9" (UID: "76267c15-7d53-4873-8328-a2ad0abd80d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:27.300231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.300130    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "76267c15-7d53-4873-8328-a2ad0abd80d9" (UID: "76267c15-7d53-4873-8328-a2ad0abd80d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:27.300231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.300141    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "76267c15-7d53-4873-8328-a2ad0abd80d9" (UID: "76267c15-7d53-4873-8328-a2ad0abd80d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:27.300231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.300216    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:27.300417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.300235    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:27.300417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.300246    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:27.302286 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.302258    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76267c15-7d53-4873-8328-a2ad0abd80d9-kube-api-access-hpghp" (OuterVolumeSpecName: "kube-api-access-hpghp") pod "76267c15-7d53-4873-8328-a2ad0abd80d9" (UID: "76267c15-7d53-4873-8328-a2ad0abd80d9"). InnerVolumeSpecName "kube-api-access-hpghp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:27.302286 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.302281    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76267c15-7d53-4873-8328-a2ad0abd80d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "76267c15-7d53-4873-8328-a2ad0abd80d9" (UID: "76267c15-7d53-4873-8328-a2ad0abd80d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:21:27.302428 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.302285    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-dshm" (OuterVolumeSpecName: "dshm") pod "76267c15-7d53-4873-8328-a2ad0abd80d9" (UID: "76267c15-7d53-4873-8328-a2ad0abd80d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:21:27.400689 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.400614    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76267c15-7d53-4873-8328-a2ad0abd80d9-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:27.400689 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.400640    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76267c15-7d53-4873-8328-a2ad0abd80d9-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:27.400689 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.400652    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpghp\" (UniqueName: \"kubernetes.io/projected/76267c15-7d53-4873-8328-a2ad0abd80d9-kube-api-access-hpghp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:27.431348 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.431312    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh" event={"ID":"217776f6-a20d-4a8e-a86e-1a432a23e689","Type":"ContainerStarted","Data":"cbd9e05d08d62496d8c84240c367a2cd5b70a52d2e4fb40145391e2a2f0b6f87"}
Sep 17 18:21:27.431491 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.431373    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh" event={"ID":"217776f6-a20d-4a8e-a86e-1a432a23e689","Type":"ContainerStarted","Data":"789e86bad693bd1c9a149f25694519f02676cbbedb72cf0bb24d2d6ffbacfb29"}
Sep 17 18:21:27.432603 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.432578    2589 generic.go:356] "Generic (PLEG): container finished" podID="76267c15-7d53-4873-8328-a2ad0abd80d9" containerID="657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb" exitCode=0
Sep 17 18:21:27.432727 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.432614    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"
Sep 17 18:21:27.432727 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.432619    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z" event={"ID":"76267c15-7d53-4873-8328-a2ad0abd80d9","Type":"ContainerDied","Data":"657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb"}
Sep 17 18:21:27.432727 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.432647    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z" event={"ID":"76267c15-7d53-4873-8328-a2ad0abd80d9","Type":"ContainerDied","Data":"69830ce6c6e105f101e6502b830a576942e6ba798bf346d7f16868b02d5baadf"}
Sep 17 18:21:27.432727 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.432669    2589 scope.go:117] "RemoveContainer" containerID="657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb"
Sep 17 18:21:27.441757 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.441738    2589 scope.go:117] "RemoveContainer" containerID="657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb"
Sep 17 18:21:27.442044 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:21:27.442024    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb\": container with ID starting with 657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb not found: ID does not exist" containerID="657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb"
Sep 17 18:21:27.442092 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.442054    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb"} err="failed to get container status \"657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb\": rpc error: code = NotFound desc = could not find container \"657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb\": container with ID starting with 657cd96ae69ba45820c5ea5789834fafcc159bc81a36157c3d603712e4d508eb not found: ID does not exist"
Sep 17 18:21:27.450548 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.450503    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh" podStartSLOduration=1.450490483 podStartE2EDuration="1.450490483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:27.450159852 +0000 UTC m=+1123.129845707" watchObservedRunningTime="2026-09-17 18:21:27.450490483 +0000 UTC m=+1123.130176345"
Sep 17 18:21:27.464283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.464253    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"]
Sep 17 18:21:27.469667 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.469643    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-83507f/explicit-model-83507f-kserve-59599f987f-7642z"]
Sep 17 18:21:27.699447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.699352    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:27.995091 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.994995    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"]
Sep 17 18:21:27.995627 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.995341    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76267c15-7d53-4873-8328-a2ad0abd80d9" containerName="main"
Sep 17 18:21:27.995627 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.995355    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="76267c15-7d53-4873-8328-a2ad0abd80d9" containerName="main"
Sep 17 18:21:27.995627 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.995461    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="76267c15-7d53-4873-8328-a2ad0abd80d9" containerName="main"
Sep 17 18:21:27.998460 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:27.998442    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.000834 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.000807    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"e2e-conflict-a-92400e60-kserve-self-signed-certs\""
Sep 17 18:21:28.004865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.004844    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-home\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.004989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.004891    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-dshm\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.004989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.004927    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj85z\" (UniqueName: \"kubernetes.io/projected/ea6562fa-9154-4fc7-9881-af4a4f9b49da-kube-api-access-rj85z\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.004989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.004967    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-model-cache\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.005333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.004991    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tls-certs\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.005333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.005210    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tmp-dir\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.008079 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.008056    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"]
Sep 17 18:21:28.033697 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.033659    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:28.105922 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.105893    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-home\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.106082 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.105937    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-dshm\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.106082 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.105959    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj85z\" (UniqueName: \"kubernetes.io/projected/ea6562fa-9154-4fc7-9881-af4a4f9b49da-kube-api-access-rj85z\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.106082 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.105989    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-model-cache\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.106082 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.106024    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tls-certs\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.106301 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.106183    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tmp-dir\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.106349 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.106317    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-home\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.106459 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.106436    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-model-cache\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.106568 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.106471    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tmp-dir\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.108372 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.108344    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-dshm\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.108673 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.108655    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tls-certs\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.113523 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.113504    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj85z\" (UniqueName: \"kubernetes.io/projected/ea6562fa-9154-4fc7-9881-af4a4f9b49da-kube-api-access-rj85z\") pod \"e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") " pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.308163 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.308136    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:28.438707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.438673    2589 generic.go:356] "Generic (PLEG): container finished" podID="217776f6-a20d-4a8e-a86e-1a432a23e689" containerID="cbd9e05d08d62496d8c84240c367a2cd5b70a52d2e4fb40145391e2a2f0b6f87" exitCode=0
Sep 17 18:21:28.438941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.438764    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh" event={"ID":"217776f6-a20d-4a8e-a86e-1a432a23e689","Type":"ContainerDied","Data":"cbd9e05d08d62496d8c84240c367a2cd5b70a52d2e4fb40145391e2a2f0b6f87"}
Sep 17 18:21:28.583051 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.582999    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"]
Sep 17 18:21:28.585330 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:21:28.585304    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea6562fa_9154_4fc7_9881_af4a4f9b49da.slice/crio-eb5b9f641a923d03ee20e41ffd321222faed6fe47cefa722d10df9efc8feca10 WatchSource:0}: Error finding container eb5b9f641a923d03ee20e41ffd321222faed6fe47cefa722d10df9efc8feca10: Status 404 returned error can't find the container with id eb5b9f641a923d03ee20e41ffd321222faed6fe47cefa722d10df9efc8feca10
Sep 17 18:21:28.634909 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.634884    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:28.683645 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:28.683611    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76267c15-7d53-4873-8328-a2ad0abd80d9" path="/var/lib/kubelet/pods/76267c15-7d53-4873-8328-a2ad0abd80d9/volumes"
Sep 17 18:21:29.360530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:29.360492    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:29.446755 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:29.446718    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948" event={"ID":"ea6562fa-9154-4fc7-9881-af4a4f9b49da","Type":"ContainerStarted","Data":"bbce35ee340da13969245d13280393551e5a24a1e2a9c5c180ecfd0503483656"}
Sep 17 18:21:29.446906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:29.446761    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948" event={"ID":"ea6562fa-9154-4fc7-9881-af4a4f9b49da","Type":"ContainerStarted","Data":"eb5b9f641a923d03ee20e41ffd321222faed6fe47cefa722d10df9efc8feca10"}
Sep 17 18:21:29.467252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:29.467204    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948" podStartSLOduration=2.467185928 podStartE2EDuration="2.467185928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:29.465446066 +0000 UTC m=+1125.145131922" watchObservedRunningTime="2026-09-17 18:21:29.467185928 +0000 UTC m=+1125.146871783"
Sep 17 18:21:29.575029 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:29.574988    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"
Sep 17 18:21:29.616665 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:29.616580    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/217776f6-a20d-4a8e-a86e-1a432a23e689-maas-api-service-ca\") pod \"217776f6-a20d-4a8e-a86e-1a432a23e689\" (UID: \"217776f6-a20d-4a8e-a86e-1a432a23e689\") "
Sep 17 18:21:29.616932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:29.616911    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/217776f6-a20d-4a8e-a86e-1a432a23e689-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "217776f6-a20d-4a8e-a86e-1a432a23e689" (UID: "217776f6-a20d-4a8e-a86e-1a432a23e689"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:29.717427 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:29.717392    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/217776f6-a20d-4a8e-a86e-1a432a23e689-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:30.331976 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:30.331934    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:30.453617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:30.453581    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh" event={"ID":"217776f6-a20d-4a8e-a86e-1a432a23e689","Type":"ContainerDied","Data":"789e86bad693bd1c9a149f25694519f02676cbbedb72cf0bb24d2d6ffbacfb29"}
Sep 17 18:21:30.453617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:30.453608    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"
Sep 17 18:21:30.454038 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:30.453632    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="789e86bad693bd1c9a149f25694519f02676cbbedb72cf0bb24d2d6ffbacfb29"
Sep 17 18:21:30.569985 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:30.569957    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:30.670819 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:30.670716    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:30.844457 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:30.844427    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:31.060714 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.060683    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:31.295681 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.295651    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:31.578195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.578162    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"]
Sep 17 18:21:31.578680 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.578662    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="217776f6-a20d-4a8e-a86e-1a432a23e689" containerName="revoke-keys"
Sep 17 18:21:31.578738 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.578685    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="217776f6-a20d-4a8e-a86e-1a432a23e689" containerName="revoke-keys"
Sep 17 18:21:31.578790 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.578780    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="217776f6-a20d-4a8e-a86e-1a432a23e689" containerName="revoke-keys"
Sep 17 18:21:31.581656 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.581639    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.584348 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.584326    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-8c963b-openshift-default-dockercfg-xz2k7\""
Sep 17 18:21:31.584464 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.584384    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 17 18:21:31.592930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.592903    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"]
Sep 17 18:21:31.605820 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.605792    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:31.632210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.632185    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-envoy\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.632321 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.632211    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-data\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.632321 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.632231    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-socket\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.632321 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.632278    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-credential-socket\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.632321 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.632295    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-podinfo\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.632321 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.632321    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-certs\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.632562 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.632347    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-token\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.632562 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.632363    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vgzd\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-kube-api-access-8vgzd\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.632562 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.632384    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f575-8e7a-4580-9462-7be5e41642a4-istiod-ca-cert\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.704202 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.704178    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:31.733679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.733656    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-token\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.733759 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.733685    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vgzd\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-kube-api-access-8vgzd\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.733759 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.733708    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f575-8e7a-4580-9462-7be5e41642a4-istiod-ca-cert\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.733856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.733777    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-envoy\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.733856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.733803    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-data\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.733856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.733830    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-socket\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.733994 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.733875    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-credential-socket\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.733994 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.733892    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-podinfo\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.733994 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.733927    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-certs\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.734311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.734289    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-data\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.734850 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.734323    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-socket\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.734973 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.734386    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-certs\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.735069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.734980    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-credential-socket\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.735125 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.735087    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f575-8e7a-4580-9462-7be5e41642a4-istiod-ca-cert\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.742484 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.742450    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-podinfo\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.743965 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.743939    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-envoy\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.744543 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.744517    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-token\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.744676 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.744657    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vgzd\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-kube-api-access-8vgzd\") pod \"e2e-noten-8c963b-openshift-default-6888687474-pv7hb\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") " pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:31.892138 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:31.892051    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:32.169604 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.169570    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"]
Sep 17 18:21:32.171074 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:21:32.171042    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fb2f575_8e7a_4580_9462_7be5e41642a4.slice/crio-53fd0eac4f0fb18b8d0531591076dacd0ac9e7237858174e79e349af8f8d55b7 WatchSource:0}: Error finding container 53fd0eac4f0fb18b8d0531591076dacd0ac9e7237858174e79e349af8f8d55b7: Status 404 returned error can't find the container with id 53fd0eac4f0fb18b8d0531591076dacd0ac9e7237858174e79e349af8f8d55b7
Sep 17 18:21:32.173356 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.173321    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:21:32.173447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.173386    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:21:32.173447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.173414    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:21:32.464323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.464237    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb" event={"ID":"5fb2f575-8e7a-4580-9462-7be5e41642a4","Type":"ContainerStarted","Data":"72ea62885a65017c27a90d709d9d032a36c10cf264170c44c896a3ea5fb6757b"}
Sep 17 18:21:32.464323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.464288    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb" event={"ID":"5fb2f575-8e7a-4580-9462-7be5e41642a4","Type":"ContainerStarted","Data":"53fd0eac4f0fb18b8d0531591076dacd0ac9e7237858174e79e349af8f8d55b7"}
Sep 17 18:21:32.486074 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.485998    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb" podStartSLOduration=1.485978917 podStartE2EDuration="1.485978917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:21:32.48560005 +0000 UTC m=+1128.165285904" watchObservedRunningTime="2026-09-17 18:21:32.485978917 +0000 UTC m=+1128.165664774"
Sep 17 18:21:32.767899 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.767864    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:32.858786 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.858757    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:32.892670 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.892639    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:32.967204 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:32.967173    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:33.897596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:33.897566    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:34.474750 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:34.474718    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:34.475567 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:34.475548    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:21:36.431485 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:36.431453    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:37.104612 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.104580    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:37.128092 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.128039    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"]
Sep 17 18:21:37.128271 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.128247    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh" podUID="18fd1ea9-1f32-443f-9114-a5b1342d5498" containerName="payload-processing" containerID="cri-o://cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f" gracePeriod=30
Sep 17 18:21:37.200443 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.200409    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:37.300654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.300620    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:37.327256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.327227    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"]
Sep 17 18:21:37.332160 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.332123    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d62f3adc-tkfzh"]
Sep 17 18:21:37.375531 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.375488    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:37.380191 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.380174    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:21:37.488293 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.488263    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd1ea9-1f32-443f-9114-a5b1342d5498-plugins-config-volume\") pod \"18fd1ea9-1f32-443f-9114-a5b1342d5498\" (UID: \"18fd1ea9-1f32-443f-9114-a5b1342d5498\") "
Sep 17 18:21:37.488704 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.488378    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g4498\" (UniqueName: \"kubernetes.io/projected/18fd1ea9-1f32-443f-9114-a5b1342d5498-kube-api-access-g4498\") pod \"18fd1ea9-1f32-443f-9114-a5b1342d5498\" (UID: \"18fd1ea9-1f32-443f-9114-a5b1342d5498\") "
Sep 17 18:21:37.488762 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.488698    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18fd1ea9-1f32-443f-9114-a5b1342d5498-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "18fd1ea9-1f32-443f-9114-a5b1342d5498" (UID: "18fd1ea9-1f32-443f-9114-a5b1342d5498"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:21:37.489809 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.489784    2589 generic.go:356] "Generic (PLEG): container finished" podID="18fd1ea9-1f32-443f-9114-a5b1342d5498" containerID="cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f" exitCode=0
Sep 17 18:21:37.489930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.489829    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"
Sep 17 18:21:37.489930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.489867    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh" event={"ID":"18fd1ea9-1f32-443f-9114-a5b1342d5498","Type":"ContainerDied","Data":"cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f"}
Sep 17 18:21:37.489930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.489925    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh" event={"ID":"18fd1ea9-1f32-443f-9114-a5b1342d5498","Type":"ContainerDied","Data":"1205068f8fc25a35c523a1930e3010f9c0c1a24492dcf160cb161ce45d9e33c0"}
Sep 17 18:21:37.490083 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.489952    2589 scope.go:117] "RemoveContainer" containerID="cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f"
Sep 17 18:21:37.490842 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.490817    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18fd1ea9-1f32-443f-9114-a5b1342d5498-kube-api-access-g4498" (OuterVolumeSpecName: "kube-api-access-g4498") pod "18fd1ea9-1f32-443f-9114-a5b1342d5498" (UID: "18fd1ea9-1f32-443f-9114-a5b1342d5498"). InnerVolumeSpecName "kube-api-access-g4498". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:21:37.502797 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.502780    2589 scope.go:117] "RemoveContainer" containerID="cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f"
Sep 17 18:21:37.503105 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:21:37.503087    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f\": container with ID starting with cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f not found: ID does not exist" containerID="cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f"
Sep 17 18:21:37.503162 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.503115    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f"} err="failed to get container status \"cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f\": rpc error: code = NotFound desc = could not find container \"cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f\": container with ID starting with cf0eba4ff97a15d0d6112cf2b12edfe8d1a0a680cf7979a5b276c66ec843389f not found: ID does not exist"
Sep 17 18:21:37.589021 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.588974    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18fd1ea9-1f32-443f-9114-a5b1342d5498-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:37.589021 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.589001    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4498\" (UniqueName: \"kubernetes.io/projected/18fd1ea9-1f32-443f-9114-a5b1342d5498-kube-api-access-g4498\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:21:37.697744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.697654    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:37.806297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.806272    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:37.811052 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.811028    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"]
Sep 17 18:21:37.817715 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:37.817693    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d62f3adc-546fbd775c-9msdh"]
Sep 17 18:21:38.308618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:38.308582    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:38.308834 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:38.308634    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:38.321612 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:38.321587    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:38.507511 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:38.507481    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:21:38.593864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:38.593777    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:38.682921 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:38.682887    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18fd1ea9-1f32-443f-9114-a5b1342d5498" path="/var/lib/kubelet/pods/18fd1ea9-1f32-443f-9114-a5b1342d5498/volumes"
Sep 17 18:21:38.683269 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:38.683255    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="217776f6-a20d-4a8e-a86e-1a432a23e689" path="/var/lib/kubelet/pods/217776f6-a20d-4a8e-a86e-1a432a23e689/volumes"
Sep 17 18:21:39.352506 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:39.352474    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:40.332405 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:40.332373    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:41.059515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:41.059476    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:41.292595 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:41.292562    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:41.393117 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:41.393033    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:43.506554 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:43.506500    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:21:43.892683 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:43.892602    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:43.989261 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:43.989221    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:44.161556 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:44.161469    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:44.497350 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:44.497260    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:44.596505 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:44.596464    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:44.696696 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:44.696668    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:48.600496 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:48.600442    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:48.812076 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:48.812043    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:49.531447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:49.531409    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:50.625949 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:50.625914    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:50.793776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:50.793730    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:50.913902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:50.913828    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:50.998178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:50.998151    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:51.128590 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:51.128557    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:57.091478 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:57.091399    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:57.309338 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:57.309310    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:57.411960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:57.411871    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:58.245811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:58.245760    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:58.559877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:58.559843    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:21:58.906392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:21:58.906303    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:01.480064 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:01.479997    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:02.087260 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:02.087221    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:02.190055 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:02.189996    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:02.301060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:02.301021    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:03.167673 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:03.167635    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:03.940298 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:03.940258    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:04.737083 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:04.737027    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:05.447790 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:05.447756    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:05.519568 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:05.519534    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:05.621914 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:05.621879    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:05.906776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:05.906744    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:22:05.943270 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:05.943233    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:06.022069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:06.022038    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:10.945709 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:10.945674    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:11.684643 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:11.684604    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:26.364707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:26.364673    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:27.818960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:27.818922    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:28.657986 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:28.657949    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:29.437512 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:29.437466    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:30.225254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:30.225217    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:30.939753 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:30.939717    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:31.704674 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:31.704646    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:32.155526 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.155489    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:32.477515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.477430    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:32.759727 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.759701    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"]
Sep 17 18:22:32.760037 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.760024    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18fd1ea9-1f32-443f-9114-a5b1342d5498" containerName="payload-processing"
Sep 17 18:22:32.760097 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.760039    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="18fd1ea9-1f32-443f-9114-a5b1342d5498" containerName="payload-processing"
Sep 17 18:22:32.760133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.760114    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="18fd1ea9-1f32-443f-9114-a5b1342d5498" containerName="payload-processing"
Sep 17 18:22:32.762925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.762886    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:32.765649 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.765624    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-03270e-openshift-default-dockercfg-2v5kh\""
Sep 17 18:22:32.778400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.778380    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"]
Sep 17 18:22:32.791496 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.791477    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:32.902055 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.901996    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-credential-socket\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:32.902224 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.902111    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-socket\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:32.902224 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.902188    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-istio-token\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:32.902323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.902224    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-certs\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:32.902323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.902250    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-envoy\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:32.902323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.902313    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0295ec94-779d-491c-a732-aba39704f88e-istiod-ca-cert\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:32.902426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.902353    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psxq2\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-kube-api-access-psxq2\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:32.902426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.902378    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-data\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:32.902426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:32.902395    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0295ec94-779d-491c-a732-aba39704f88e-istio-podinfo\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003554 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003518    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-socket\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003567    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-istio-token\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003592    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-certs\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003626    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-envoy\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003673    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0295ec94-779d-491c-a732-aba39704f88e-istiod-ca-cert\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003714    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psxq2\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-kube-api-access-psxq2\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003737    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-data\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003761    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0295ec94-779d-491c-a732-aba39704f88e-istio-podinfo\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003800    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-credential-socket\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.003989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.003977    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-socket\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.004213 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.004192    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-certs\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.004297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.004261    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-data\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.004345 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.004289    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-credential-socket\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.004481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.004458    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0295ec94-779d-491c-a732-aba39704f88e-istiod-ca-cert\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.006102 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.006084    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-envoy\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.006405 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.006385    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0295ec94-779d-491c-a732-aba39704f88e-istio-podinfo\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.012457 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.012404    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psxq2\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-kube-api-access-psxq2\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.012539 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.012470    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-istio-token\") pod \"e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") " pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.072699 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.072668    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:33.113604 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.113566    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:33.365827 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.365803    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"]
Sep 17 18:22:33.367462 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:22:33.367436    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0295ec94_779d_491c_a732_aba39704f88e.slice/crio-0aabbf420c9ee71ea85e51cac0ef944599803d706e7830aa65e8537e62da619e WatchSource:0}: Error finding container 0aabbf420c9ee71ea85e51cac0ef944599803d706e7830aa65e8537e62da619e: Status 404 returned error can't find the container with id 0aabbf420c9ee71ea85e51cac0ef944599803d706e7830aa65e8537e62da619e
Sep 17 18:22:33.369780 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.369729    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:22:33.369880 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.369828    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:22:33.369880 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.369872    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:22:33.772656 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.772619    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" event={"ID":"0295ec94-779d-491c-a732-aba39704f88e","Type":"ContainerStarted","Data":"cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42"}
Sep 17 18:22:33.772854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.772713    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" event={"ID":"0295ec94-779d-491c-a732-aba39704f88e","Type":"ContainerStarted","Data":"0aabbf420c9ee71ea85e51cac0ef944599803d706e7830aa65e8537e62da619e"}
Sep 17 18:22:33.805307 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:33.805250    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" podStartSLOduration=1.8052344850000002 podStartE2EDuration="1.805234485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:22:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:22:33.803470883 +0000 UTC m=+1189.483156749" watchObservedRunningTime="2026-09-17 18:22:33.805234485 +0000 UTC m=+1189.484920386"
Sep 17 18:22:34.073437 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:34.073350    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:34.078727 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:34.078704    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:34.701445 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:34.701415    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:22:34.777713 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:34.777683    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:34.778629 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:34.778608    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:22:35.104877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:35.104847    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:35.727019 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:35.726967    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:35.828107 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:35.828074    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:35.926412 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:35.926374    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:36.845731 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:36.845690    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:37.586068 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:37.586036    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:38.289133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:38.289099    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:39.141976 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:39.141925    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:40.040116 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:40.040081    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:40.142913 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:40.142881    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:40.202793 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:40.202759    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:40.521104 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:40.521076    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:40.608352 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:40.608315    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:45.602896 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:45.602842    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"]
Sep 17 18:22:45.603625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:45.603123    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb" podUID="5fb2f575-8e7a-4580-9462-7be5e41642a4" containerName="istio-proxy" containerID="cri-o://72ea62885a65017c27a90d709d9d032a36c10cf264170c44c896a3ea5fb6757b" gracePeriod=30
Sep 17 18:22:45.626908 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:45.626874    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:45.852749 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:45.852710    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:45.956975 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:45.956875    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:46.633380 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:46.633345    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:22:46.633913 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:46.633895    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:22:47.580500 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:47.580473    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"]
Sep 17 18:22:47.607423 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:47.607396    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:49.477475 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:49.477438    2589 patch_prober.go:28] interesting pod/e2e-noten-8c963b-openshift-default-6888687474-pv7hb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 18:22:49.477875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:49.477504    2589 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb" podUID="5fb2f575-8e7a-4580-9462-7be5e41642a4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:22:49.595974 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:49.595943    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:50.858199 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.858149    2589 generic.go:356] "Generic (PLEG): container finished" podID="5fb2f575-8e7a-4580-9462-7be5e41642a4" containerID="72ea62885a65017c27a90d709d9d032a36c10cf264170c44c896a3ea5fb6757b" exitCode=0
Sep 17 18:22:50.858596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.858224    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb" event={"ID":"5fb2f575-8e7a-4580-9462-7be5e41642a4","Type":"ContainerDied","Data":"72ea62885a65017c27a90d709d9d032a36c10cf264170c44c896a3ea5fb6757b"}
Sep 17 18:22:50.858596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.858291    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb" event={"ID":"5fb2f575-8e7a-4580-9462-7be5e41642a4","Type":"ContainerDied","Data":"53fd0eac4f0fb18b8d0531591076dacd0ac9e7237858174e79e349af8f8d55b7"}
Sep 17 18:22:50.858596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.858316    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53fd0eac4f0fb18b8d0531591076dacd0ac9e7237858174e79e349af8f8d55b7"
Sep 17 18:22:50.873048 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.872988    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:22:50.969851 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.969821    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-envoy\") pod \"5fb2f575-8e7a-4580-9462-7be5e41642a4\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") "
Sep 17 18:22:50.969851 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.969852    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-data\") pod \"5fb2f575-8e7a-4580-9462-7be5e41642a4\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") "
Sep 17 18:22:50.970135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.969883    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-certs\") pod \"5fb2f575-8e7a-4580-9462-7be5e41642a4\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") "
Sep 17 18:22:50.970135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.969903    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-podinfo\") pod \"5fb2f575-8e7a-4580-9462-7be5e41642a4\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") "
Sep 17 18:22:50.970135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.969931    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f575-8e7a-4580-9462-7be5e41642a4-istiod-ca-cert\") pod \"5fb2f575-8e7a-4580-9462-7be5e41642a4\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") "
Sep 17 18:22:50.970135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.969956    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-credential-socket\") pod \"5fb2f575-8e7a-4580-9462-7be5e41642a4\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") "
Sep 17 18:22:50.970135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.969980    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-socket\") pod \"5fb2f575-8e7a-4580-9462-7be5e41642a4\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") "
Sep 17 18:22:50.970135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.970045    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-token\") pod \"5fb2f575-8e7a-4580-9462-7be5e41642a4\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") "
Sep 17 18:22:50.970135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.970070    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8vgzd\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-kube-api-access-8vgzd\") pod \"5fb2f575-8e7a-4580-9462-7be5e41642a4\" (UID: \"5fb2f575-8e7a-4580-9462-7be5e41642a4\") "
Sep 17 18:22:50.970450 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.970142    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-data" (OuterVolumeSpecName: "istio-data") pod "5fb2f575-8e7a-4580-9462-7be5e41642a4" (UID: "5fb2f575-8e7a-4580-9462-7be5e41642a4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:50.970450 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.970168    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5fb2f575-8e7a-4580-9462-7be5e41642a4" (UID: "5fb2f575-8e7a-4580-9462-7be5e41642a4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:50.970450 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.970386    2589 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:22:50.970450 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.970407    2589 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:22:50.970660 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.970532    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5fb2f575-8e7a-4580-9462-7be5e41642a4" (UID: "5fb2f575-8e7a-4580-9462-7be5e41642a4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:50.970885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.970695    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5fb2f575-8e7a-4580-9462-7be5e41642a4" (UID: "5fb2f575-8e7a-4580-9462-7be5e41642a4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:50.970885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.970773    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fb2f575-8e7a-4580-9462-7be5e41642a4-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5fb2f575-8e7a-4580-9462-7be5e41642a4" (UID: "5fb2f575-8e7a-4580-9462-7be5e41642a4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:22:50.972374 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.972349    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-token" (OuterVolumeSpecName: "istio-token") pod "5fb2f575-8e7a-4580-9462-7be5e41642a4" (UID: "5fb2f575-8e7a-4580-9462-7be5e41642a4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:50.972497 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.972397    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5fb2f575-8e7a-4580-9462-7be5e41642a4" (UID: "5fb2f575-8e7a-4580-9462-7be5e41642a4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:22:50.972714 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.972687    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5fb2f575-8e7a-4580-9462-7be5e41642a4" (UID: "5fb2f575-8e7a-4580-9462-7be5e41642a4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:22:50.972714 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:50.972686    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-kube-api-access-8vgzd" (OuterVolumeSpecName: "kube-api-access-8vgzd") pod "5fb2f575-8e7a-4580-9462-7be5e41642a4" (UID: "5fb2f575-8e7a-4580-9462-7be5e41642a4"). InnerVolumeSpecName "kube-api-access-8vgzd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:22:51.042987 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.042955    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:22:51.071694 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.071668    2589 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.071694 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.071694    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vgzd\" (UniqueName: \"kubernetes.io/projected/5fb2f575-8e7a-4580-9462-7be5e41642a4-kube-api-access-8vgzd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.071868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.071709    2589 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.071868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.071721    2589 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5fb2f575-8e7a-4580-9462-7be5e41642a4-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.071868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.071732    2589 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5fb2f575-8e7a-4580-9462-7be5e41642a4-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.071868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.071744    2589 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.071868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.071755    2589 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5fb2f575-8e7a-4580-9462-7be5e41642a4-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:22:51.862773 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.862743    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"
Sep 17 18:22:51.884632 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.884592    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"]
Sep 17 18:22:51.891840 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:51.891802    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-8c963b-openshift-default-6888687474-pv7hb"]
Sep 17 18:22:52.685351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:52.685307    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fb2f575-8e7a-4580-9462-7be5e41642a4" path="/var/lib/kubelet/pods/5fb2f575-8e7a-4580-9462-7be5e41642a4/volumes"
Sep 17 18:22:58.898712 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:58.898676    2589 generic.go:356] "Generic (PLEG): container finished" podID="0295ec94-779d-491c-a732-aba39704f88e" containerID="cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42" exitCode=0
Sep 17 18:22:58.899117 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:58.898750    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" event={"ID":"0295ec94-779d-491c-a732-aba39704f88e","Type":"ContainerDied","Data":"cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42"}
Sep 17 18:22:58.899218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:58.899201    2589 scope.go:117] "RemoveContainer" containerID="cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42"
Sep 17 18:22:58.900484 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:58.900455    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:22:58.900560 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:58.900537    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:22:58.900597 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:58.900570    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:22:59.912903 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:22:59.912870    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" event={"ID":"0295ec94-779d-491c-a732-aba39704f88e","Type":"ContainerStarted","Data":"0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea"}
Sep 17 18:23:00.073870 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:00.073835    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:23:00.075082 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:00.075061    2589 patch_prober.go:28] interesting pod/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 17 18:23:00.075138 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:00.075121    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 17 18:23:01.073792 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.073757    2589 patch_prober.go:28] interesting pod/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 17 18:23:01.074197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.073814    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 17 18:23:01.822291 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.822261    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"]
Sep 17 18:23:01.822645 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.822632    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fb2f575-8e7a-4580-9462-7be5e41642a4" containerName="istio-proxy"
Sep 17 18:23:01.822692 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.822647    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fb2f575-8e7a-4580-9462-7be5e41642a4" containerName="istio-proxy"
Sep 17 18:23:01.822727 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.822706    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fb2f575-8e7a-4580-9462-7be5e41642a4" containerName="istio-proxy"
Sep 17 18:23:01.824596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.824580    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"
Sep 17 18:23:01.827039 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.826993    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:23:01.827177 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.827129    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:23:01.833140 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.833117    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"]
Sep 17 18:23:01.895030 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.890469    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370aeb40-2a31-4f94-93a8-7a9cfb70e0b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9\" (UID: \"370aeb40-2a31-4f94-93a8-7a9cfb70e0b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"
Sep 17 18:23:01.992815 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.991914    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370aeb40-2a31-4f94-93a8-7a9cfb70e0b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9\" (UID: \"370aeb40-2a31-4f94-93a8-7a9cfb70e0b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"
Sep 17 18:23:01.993039 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:01.992768    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370aeb40-2a31-4f94-93a8-7a9cfb70e0b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9\" (UID: \"370aeb40-2a31-4f94-93a8-7a9cfb70e0b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"
Sep 17 18:23:02.073101 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.072991    2589 patch_prober.go:28] interesting pod/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 17 18:23:02.073101 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.073078    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 17 18:23:02.136456 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.136426    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"
Sep 17 18:23:02.154367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.154321    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:02.423388 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.423364    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"]
Sep 17 18:23:02.425463 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:23:02.425435    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod370aeb40_2a31_4f94_93a8_7a9cfb70e0b8.slice/crio-63609669da98be8de33ab421404b4bd1fa7556e06310b45d084aeeb875714015 WatchSource:0}: Error finding container 63609669da98be8de33ab421404b4bd1fa7556e06310b45d084aeeb875714015: Status 404 returned error can't find the container with id 63609669da98be8de33ab421404b4bd1fa7556e06310b45d084aeeb875714015
Sep 17 18:23:02.602432 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.602383    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk" podUID="f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" containerName="main" containerID="cri-o://ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f" gracePeriod=45
Sep 17 18:23:02.857333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.857309    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:23:02.917633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.917602    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsrch\" (UniqueName: \"kubernetes.io/projected/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-kube-api-access-zsrch\") pod \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") "
Sep 17 18:23:02.917633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.917637    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-home\") pod \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") "
Sep 17 18:23:02.917878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.917657    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-dshm\") pod \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") "
Sep 17 18:23:02.917878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.917713    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tmp-dir\") pod \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") "
Sep 17 18:23:02.917878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.917732    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-model-cache\") pod \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") "
Sep 17 18:23:02.917878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.917765    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tls-certs\") pod \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\" (UID: \"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4\") "
Sep 17 18:23:02.918127 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.917904    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-home" (OuterVolumeSpecName: "home") pod "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" (UID: "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:02.918127 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.917979    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" (UID: "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:02.918127 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.918035    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-model-cache" (OuterVolumeSpecName: "model-cache") pod "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" (UID: "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:02.918127 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.918068    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:02.920110 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.920083    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-dshm" (OuterVolumeSpecName: "dshm") pod "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" (UID: "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:23:02.920216 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.920108    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" (UID: "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:23:02.920216 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.920170    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-kube-api-access-zsrch" (OuterVolumeSpecName: "kube-api-access-zsrch") pod "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" (UID: "f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4"). InnerVolumeSpecName "kube-api-access-zsrch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:02.935336 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.935277    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9" event={"ID":"370aeb40-2a31-4f94-93a8-7a9cfb70e0b8","Type":"ContainerStarted","Data":"0db121c02a29929c46134925f46db627cb6a596bbec9c4d671f4db49e83e1cd2"}
Sep 17 18:23:02.935473 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.935406    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9" event={"ID":"370aeb40-2a31-4f94-93a8-7a9cfb70e0b8","Type":"ContainerStarted","Data":"63609669da98be8de33ab421404b4bd1fa7556e06310b45d084aeeb875714015"}
Sep 17 18:23:02.935733 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.935710    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:02.936753 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.936726    2589 generic.go:356] "Generic (PLEG): container finished" podID="f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" containerID="ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f" exitCode=0
Sep 17 18:23:02.936865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.936749    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk" event={"ID":"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4","Type":"ContainerDied","Data":"ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f"}
Sep 17 18:23:02.936865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.936764    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"
Sep 17 18:23:02.936865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.936782    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk" event={"ID":"f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4","Type":"ContainerDied","Data":"710c7208d3eb666508ccd422b14a00a7696c1374017263d39912f8ed0578a2d2"}
Sep 17 18:23:02.936865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.936798    2589 scope.go:117] "RemoveContainer" containerID="ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f"
Sep 17 18:23:02.946141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.946124    2589 scope.go:117] "RemoveContainer" containerID="ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f"
Sep 17 18:23:02.946396 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:02.946378    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f\": container with ID starting with ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f not found: ID does not exist" containerID="ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f"
Sep 17 18:23:02.946436 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.946404    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f"} err="failed to get container status \"ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f\": rpc error: code = NotFound desc = could not find container \"ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f\": container with ID starting with ec2ae815bd71967a0d24ec29b74db1f473784e6b731fc49f85d8d8ca5962b30f not found: ID does not exist"
Sep 17 18:23:02.950711 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.950662    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9" podStartSLOduration=1.9506508729999998 podStartE2EDuration="1.950650873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:02.950063301 +0000 UTC m=+1218.629749154" watchObservedRunningTime="2026-09-17 18:23:02.950650873 +0000 UTC m=+1218.630336761"
Sep 17 18:23:02.963735 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.963710    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"]
Sep 17 18:23:02.967935 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:02.967886    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c31adf/test-model-c31adf-kserve-5b98bb7977-lqvnk"]
Sep 17 18:23:03.018945 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.018919    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:03.018945 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.018942    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsrch\" (UniqueName: \"kubernetes.io/projected/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-kube-api-access-zsrch\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:03.019150 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.018953    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:03.019150 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.018964    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:03.019150 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.018975    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:03.074128 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.074098    2589 patch_prober.go:28] interesting pod/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 17 18:23:03.074275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.074159    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 17 18:23:03.743619 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.743589    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:03.942872 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.942835    2589 generic.go:356] "Generic (PLEG): container finished" podID="370aeb40-2a31-4f94-93a8-7a9cfb70e0b8" containerID="0db121c02a29929c46134925f46db627cb6a596bbec9c4d671f4db49e83e1cd2" exitCode=0
Sep 17 18:23:03.943091 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:03.942911    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9" event={"ID":"370aeb40-2a31-4f94-93a8-7a9cfb70e0b8","Type":"ContainerDied","Data":"0db121c02a29929c46134925f46db627cb6a596bbec9c4d671f4db49e83e1cd2"}
Sep 17 18:23:04.073553 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:04.073518    2589 patch_prober.go:28] interesting pod/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused" start-of-body=
Sep 17 18:23:04.073729 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:04.073576    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.53:15021/healthz/ready\": dial tcp 10.132.0.53:15021: connect: connection refused"
Sep 17 18:23:04.529238 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:04.529201    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:04.628578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:04.628538    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:04.708955 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:04.707866    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" path="/var/lib/kubelet/pods/f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4/volumes"
Sep 17 18:23:04.778095 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:04.778064    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:23:05.091032 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.090982    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"
Sep 17 18:23:05.099444 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.099415    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:23:05.101696 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.101670    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:23:05.231731 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.231644    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:05.239679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.239659    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370aeb40-2a31-4f94-93a8-7a9cfb70e0b8-maas-api-service-ca\") pod \"370aeb40-2a31-4f94-93a8-7a9cfb70e0b8\" (UID: \"370aeb40-2a31-4f94-93a8-7a9cfb70e0b8\") "
Sep 17 18:23:05.240111 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.240082    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/370aeb40-2a31-4f94-93a8-7a9cfb70e0b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "370aeb40-2a31-4f94-93a8-7a9cfb70e0b8" (UID: "370aeb40-2a31-4f94-93a8-7a9cfb70e0b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:05.340748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.340714    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/370aeb40-2a31-4f94-93a8-7a9cfb70e0b8-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:05.723548 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.723515    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:05.954561 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.954533    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"
Sep 17 18:23:05.955065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.954577    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9" event={"ID":"370aeb40-2a31-4f94-93a8-7a9cfb70e0b8","Type":"ContainerDied","Data":"63609669da98be8de33ab421404b4bd1fa7556e06310b45d084aeeb875714015"}
Sep 17 18:23:05.955065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:05.954626    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63609669da98be8de33ab421404b4bd1fa7556e06310b45d084aeeb875714015"
Sep 17 18:23:06.048993 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.048955    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:06.424593 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.424497    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7628-b7f6ee"]
Sep 17 18:23:06.424917 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.424899    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" containerName="main"
Sep 17 18:23:06.424996 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.424920    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" containerName="main"
Sep 17 18:23:06.424996 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.424931    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="370aeb40-2a31-4f94-93a8-7a9cfb70e0b8" containerName="revoke-keys"
Sep 17 18:23:06.424996 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.424939    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="370aeb40-2a31-4f94-93a8-7a9cfb70e0b8" containerName="revoke-keys"
Sep 17 18:23:06.425183 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.425061    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="f32aa8f2-ecc8-43ef-8fb3-67e11edf43a4" containerName="main"
Sep 17 18:23:06.425183 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.425081    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="370aeb40-2a31-4f94-93a8-7a9cfb70e0b8" containerName="revoke-keys"
Sep 17 18:23:06.427225 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.427205    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b7f6ee"
Sep 17 18:23:06.429538 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.429522    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s5zgf\""
Sep 17 18:23:06.432218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.432197    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-b7f6ee"]
Sep 17 18:23:06.550911 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.550876    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78ktb\" (UniqueName: \"kubernetes.io/projected/ba3ce991-55df-4e46-908b-eaf1ff305c0d-kube-api-access-78ktb\") pod \"test-curl-7628-b7f6ee\" (UID: \"ba3ce991-55df-4e46-908b-eaf1ff305c0d\") " pod="opendatahub/test-curl-7628-b7f6ee"
Sep 17 18:23:06.651935 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.651900    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78ktb\" (UniqueName: \"kubernetes.io/projected/ba3ce991-55df-4e46-908b-eaf1ff305c0d-kube-api-access-78ktb\") pod \"test-curl-7628-b7f6ee\" (UID: \"ba3ce991-55df-4e46-908b-eaf1ff305c0d\") " pod="opendatahub/test-curl-7628-b7f6ee"
Sep 17 18:23:06.660414 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.660386    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-78ktb\" (UniqueName: \"kubernetes.io/projected/ba3ce991-55df-4e46-908b-eaf1ff305c0d-kube-api-access-78ktb\") pod \"test-curl-7628-b7f6ee\" (UID: \"ba3ce991-55df-4e46-908b-eaf1ff305c0d\") " pod="opendatahub/test-curl-7628-b7f6ee"
Sep 17 18:23:06.696600 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.696535    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:23:06.737220 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.737195    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b7f6ee"
Sep 17 18:23:06.762316 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:06.762291    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:07.005823 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:07.005797    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-b7f6ee"]
Sep 17 18:23:07.008148 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:23:07.008122    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba3ce991_55df_4e46_908b_eaf1ff305c0d.slice/crio-648ca678de066d4d1f2ec9c934cb2bbd9e356cffcc8d31237a715c8da8bf4247 WatchSource:0}: Error finding container 648ca678de066d4d1f2ec9c934cb2bbd9e356cffcc8d31237a715c8da8bf4247: Status 404 returned error can't find the container with id 648ca678de066d4d1f2ec9c934cb2bbd9e356cffcc8d31237a715c8da8bf4247
Sep 17 18:23:07.553248 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:07.553210    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:07.966348 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:07.966251    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-b7f6ee" event={"ID":"ba3ce991-55df-4e46-908b-eaf1ff305c0d","Type":"ContainerStarted","Data":"2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3"}
Sep 17 18:23:07.966348 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:07.966299    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-b7f6ee" event={"ID":"ba3ce991-55df-4e46-908b-eaf1ff305c0d","Type":"ContainerStarted","Data":"648ca678de066d4d1f2ec9c934cb2bbd9e356cffcc8d31237a715c8da8bf4247"}
Sep 17 18:23:07.986123 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:07.986064    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7628-b7f6ee" podStartSLOduration=1.98604638 podStartE2EDuration="1.98604638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:07.984390959 +0000 UTC m=+1223.664076812" watchObservedRunningTime="2026-09-17 18:23:07.98604638 +0000 UTC m=+1223.665732233"
Sep 17 18:23:08.566585 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.566547    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7628-b7f6ee"]
Sep 17 18:23:08.571811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.571783    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7628-b7f6ee"]
Sep 17 18:23:08.684128 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.684054    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7628-edab31"]
Sep 17 18:23:08.684369 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.684357    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3ce991-55df-4e46-908b-eaf1ff305c0d" containerName="test-curl-7628-b7f6ee"
Sep 17 18:23:08.684412 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.684371    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3ce991-55df-4e46-908b-eaf1ff305c0d" containerName="test-curl-7628-b7f6ee"
Sep 17 18:23:08.684460 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.684451    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3ce991-55df-4e46-908b-eaf1ff305c0d" containerName="test-curl-7628-b7f6ee"
Sep 17 18:23:08.692878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.692845    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-edab31"
Sep 17 18:23:08.694179 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.694151    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-edab31"]
Sep 17 18:23:08.773506 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.773476    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgvtb\" (UniqueName: \"kubernetes.io/projected/44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8-kube-api-access-mgvtb\") pod \"test-curl-7628-edab31\" (UID: \"44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8\") " pod="opendatahub/test-curl-7628-edab31"
Sep 17 18:23:08.874040 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.873989    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgvtb\" (UniqueName: \"kubernetes.io/projected/44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8-kube-api-access-mgvtb\") pod \"test-curl-7628-edab31\" (UID: \"44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8\") " pod="opendatahub/test-curl-7628-edab31"
Sep 17 18:23:08.883204 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:08.883178    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgvtb\" (UniqueName: \"kubernetes.io/projected/44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8-kube-api-access-mgvtb\") pod \"test-curl-7628-edab31\" (UID: \"44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8\") " pod="opendatahub/test-curl-7628-edab31"
Sep 17 18:23:08.971246 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:08.971177    2589 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7628-b7f6ee" container="test-curl-7628-b7f6ee"
Sep 17 18:23:09.004623 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:09.004598    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-edab31"
Sep 17 18:23:09.271995 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:09.271967    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-edab31"]
Sep 17 18:23:09.275590 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:23:09.275566    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44c77a60_1f99_4eb7_b7ee_c928b2a8e6f8.slice/crio-fbd36c19a54aef8034fd0ab827f56981249a57766a5bc05b2967f0e49b4f115d WatchSource:0}: Error finding container fbd36c19a54aef8034fd0ab827f56981249a57766a5bc05b2967f0e49b4f115d: Status 404 returned error can't find the container with id fbd36c19a54aef8034fd0ab827f56981249a57766a5bc05b2967f0e49b4f115d
Sep 17 18:23:09.977441 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:09.977404    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-edab31" event={"ID":"44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8","Type":"ContainerStarted","Data":"aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00"}
Sep 17 18:23:09.977441 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:09.977449    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-edab31" event={"ID":"44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8","Type":"ContainerStarted","Data":"fbd36c19a54aef8034fd0ab827f56981249a57766a5bc05b2967f0e49b4f115d"}
Sep 17 18:23:09.977912 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:09.977493    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7628-b7f6ee" podUID="ba3ce991-55df-4e46-908b-eaf1ff305c0d" containerName="test-curl-7628-b7f6ee" containerID="cri-o://2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3" gracePeriod=2
Sep 17 18:23:09.992451 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:09.992406    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7628-edab31" podStartSLOduration=1.992392486 podStartE2EDuration="1.992392486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:09.991787102 +0000 UTC m=+1225.671472956" watchObservedRunningTime="2026-09-17 18:23:09.992392486 +0000 UTC m=+1225.672078339"
Sep 17 18:23:10.582803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.582773    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7628-edab31"]
Sep 17 18:23:10.587217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.587188    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7628-edab31"]
Sep 17 18:23:10.706290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.706214    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7628-e13389"]
Sep 17 18:23:10.706660 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.706642    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8" containerName="test-curl-7628-edab31"
Sep 17 18:23:10.706724 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.706664    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8" containerName="test-curl-7628-edab31"
Sep 17 18:23:10.706763 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.706754    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8" containerName="test-curl-7628-edab31"
Sep 17 18:23:10.709210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.709193    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-e13389"
Sep 17 18:23:10.716404 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.716382    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-e13389"]
Sep 17 18:23:10.891815 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.891780    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkbtt\" (UniqueName: \"kubernetes.io/projected/f0a443cb-b33f-4121-a880-02b1a5939e8b-kube-api-access-rkbtt\") pod \"test-curl-7628-e13389\" (UID: \"f0a443cb-b33f-4121-a880-02b1a5939e8b\") " pod="opendatahub/test-curl-7628-e13389"
Sep 17 18:23:10.982165 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:10.982086    2589 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7628-edab31" container="test-curl-7628-edab31"
Sep 17 18:23:10.992319 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:10.992291    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkbtt\" (UniqueName: \"kubernetes.io/projected/f0a443cb-b33f-4121-a880-02b1a5939e8b-kube-api-access-rkbtt\") pod \"test-curl-7628-e13389\" (UID: \"f0a443cb-b33f-4121-a880-02b1a5939e8b\") " pod="opendatahub/test-curl-7628-e13389"
Sep 17 18:23:11.000811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:11.000789    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkbtt\" (UniqueName: \"kubernetes.io/projected/f0a443cb-b33f-4121-a880-02b1a5939e8b-kube-api-access-rkbtt\") pod \"test-curl-7628-e13389\" (UID: \"f0a443cb-b33f-4121-a880-02b1a5939e8b\") " pod="opendatahub/test-curl-7628-e13389"
Sep 17 18:23:11.018567 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:11.018546    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-e13389"
Sep 17 18:23:11.289136 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:11.289111    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-e13389"]
Sep 17 18:23:11.291745 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:23:11.291706    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0a443cb_b33f_4121_a880_02b1a5939e8b.slice/crio-93d4544972e6ec8d647fba986603f7fd71c0aee487d763d144a968d7726e0bbf WatchSource:0}: Error finding container 93d4544972e6ec8d647fba986603f7fd71c0aee487d763d144a968d7726e0bbf: Status 404 returned error can't find the container with id 93d4544972e6ec8d647fba986603f7fd71c0aee487d763d144a968d7726e0bbf
Sep 17 18:23:11.967122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:11.967080    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:11.988075 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:11.988043    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-e13389" event={"ID":"f0a443cb-b33f-4121-a880-02b1a5939e8b","Type":"ContainerStarted","Data":"43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d"}
Sep 17 18:23:11.988991 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:11.988093    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-e13389" event={"ID":"f0a443cb-b33f-4121-a880-02b1a5939e8b","Type":"ContainerStarted","Data":"93d4544972e6ec8d647fba986603f7fd71c0aee487d763d144a968d7726e0bbf"}
Sep 17 18:23:11.988991 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:11.988236    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7628-edab31" podUID="44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8" containerName="test-curl-7628-edab31" containerID="cri-o://aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00" gracePeriod=2
Sep 17 18:23:12.003198 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.003151    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7628-e13389" podStartSLOduration=2.003131687 podStartE2EDuration="2.003131687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:12.002481344 +0000 UTC m=+1227.682167198" watchObservedRunningTime="2026-09-17 18:23:12.003131687 +0000 UTC m=+1227.682817542"
Sep 17 18:23:12.073862 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.073835    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:12.114105 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.114084    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b7f6ee"
Sep 17 18:23:12.203125 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.203099    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-78ktb\" (UniqueName: \"kubernetes.io/projected/ba3ce991-55df-4e46-908b-eaf1ff305c0d-kube-api-access-78ktb\") pod \"ba3ce991-55df-4e46-908b-eaf1ff305c0d\" (UID: \"ba3ce991-55df-4e46-908b-eaf1ff305c0d\") "
Sep 17 18:23:12.205141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.205116    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3ce991-55df-4e46-908b-eaf1ff305c0d-kube-api-access-78ktb" (OuterVolumeSpecName: "kube-api-access-78ktb") pod "ba3ce991-55df-4e46-908b-eaf1ff305c0d" (UID: "ba3ce991-55df-4e46-908b-eaf1ff305c0d"). InnerVolumeSpecName "kube-api-access-78ktb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:12.304491 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.304461    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78ktb\" (UniqueName: \"kubernetes.io/projected/ba3ce991-55df-4e46-908b-eaf1ff305c0d-kube-api-access-78ktb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:12.573918 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.573838    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:12.592941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.592909    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7628-e13389"]
Sep 17 18:23:12.595594 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.595562    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7628-e13389"]
Sep 17 18:23:12.673360 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.673325    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:12.683470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.683434    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3ce991-55df-4e46-908b-eaf1ff305c0d" path="/var/lib/kubelet/pods/ba3ce991-55df-4e46-908b-eaf1ff305c0d/volumes"
Sep 17 18:23:12.707696 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.707671    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7628-cad226"]
Sep 17 18:23:12.707992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.707981    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0a443cb-b33f-4121-a880-02b1a5939e8b" containerName="test-curl-7628-e13389"
Sep 17 18:23:12.708053 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.707994    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0a443cb-b33f-4121-a880-02b1a5939e8b" containerName="test-curl-7628-e13389"
Sep 17 18:23:12.708087 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.708065    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0a443cb-b33f-4121-a880-02b1a5939e8b" containerName="test-curl-7628-e13389"
Sep 17 18:23:12.709964 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.709951    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-cad226"
Sep 17 18:23:12.718327 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.718303    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-cad226"]
Sep 17 18:23:12.773974 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.773939    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:12.799862 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.799832    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"]
Sep 17 18:23:12.802913 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.802891    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c31adf-rt6j9"]
Sep 17 18:23:12.807528 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.807508    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bs7c\" (UniqueName: \"kubernetes.io/projected/877c7515-e3fe-4cb8-997a-09ba340625a0-kube-api-access-8bs7c\") pod \"test-curl-7628-cad226\" (UID: \"877c7515-e3fe-4cb8-997a-09ba340625a0\") " pod="opendatahub/test-curl-7628-cad226"
Sep 17 18:23:12.831830 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.831765    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:12.908550 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.908521    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bs7c\" (UniqueName: \"kubernetes.io/projected/877c7515-e3fe-4cb8-997a-09ba340625a0-kube-api-access-8bs7c\") pod \"test-curl-7628-cad226\" (UID: \"877c7515-e3fe-4cb8-997a-09ba340625a0\") " pod="opendatahub/test-curl-7628-cad226"
Sep 17 18:23:12.917192 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.917168    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bs7c\" (UniqueName: \"kubernetes.io/projected/877c7515-e3fe-4cb8-997a-09ba340625a0-kube-api-access-8bs7c\") pod \"test-curl-7628-cad226\" (UID: \"877c7515-e3fe-4cb8-997a-09ba340625a0\") " pod="opendatahub/test-curl-7628-cad226"
Sep 17 18:23:12.994601 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.994562    2589 generic.go:356] "Generic (PLEG): container finished" podID="ba3ce991-55df-4e46-908b-eaf1ff305c0d" containerID="2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3" exitCode=137
Sep 17 18:23:12.994989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.994624    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-b7f6ee"
Sep 17 18:23:12.994989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:12.994660    2589 scope.go:117] "RemoveContainer" containerID="2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3"
Sep 17 18:23:12.994989 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:12.994746    2589 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7628-e13389" container="test-curl-7628-e13389"
Sep 17 18:23:13.003189 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:13.003173    2589 scope.go:117] "RemoveContainer" containerID="2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3"
Sep 17 18:23:13.003466 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:13.003447    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3\": container with ID starting with 2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3 not found: ID does not exist" containerID="2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3"
Sep 17 18:23:13.003523 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:13.003473    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3"} err="failed to get container status \"2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3\": rpc error: code = NotFound desc = could not find container \"2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3\": container with ID starting with 2a4b410e69b5d10d6cdf5b69f90f933346cf0d23dbe815af53c8166933c2ecb3 not found: ID does not exist"
Sep 17 18:23:13.019107 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:13.019087    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-cad226"
Sep 17 18:23:13.136801 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:13.136712    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:13.289181 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:13.289158    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7628-cad226"]
Sep 17 18:23:13.291572 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:23:13.291545    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod877c7515_e3fe_4cb8_997a_09ba340625a0.slice/crio-d4ce442d5dca2bd2ba5e67706150b239148abb74765b364608d6f440c2b793cc WatchSource:0}: Error finding container d4ce442d5dca2bd2ba5e67706150b239148abb74765b364608d6f440c2b793cc: Status 404 returned error can't find the container with id d4ce442d5dca2bd2ba5e67706150b239148abb74765b364608d6f440c2b793cc
Sep 17 18:23:13.397950 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:13.397872    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:14.001524 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.001496    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-cad226" event={"ID":"877c7515-e3fe-4cb8-997a-09ba340625a0","Type":"ContainerStarted","Data":"21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58"}
Sep 17 18:23:14.001945 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.001540    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7628-cad226" event={"ID":"877c7515-e3fe-4cb8-997a-09ba340625a0","Type":"ContainerStarted","Data":"d4ce442d5dca2bd2ba5e67706150b239148abb74765b364608d6f440c2b793cc"}
Sep 17 18:23:14.001945 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.001545    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7628-e13389" podUID="f0a443cb-b33f-4121-a880-02b1a5939e8b" containerName="test-curl-7628-e13389" containerID="cri-o://43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d" gracePeriod=2
Sep 17 18:23:14.016481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.016431    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7628-cad226" podStartSLOduration=2.016412478 podStartE2EDuration="2.016412478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:14.015904036 +0000 UTC m=+1229.695589889" watchObservedRunningTime="2026-09-17 18:23:14.016412478 +0000 UTC m=+1229.696098333"
Sep 17 18:23:14.139188 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.139165    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-edab31"
Sep 17 18:23:14.156185 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.156157    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:14.320267 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.320231    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mgvtb\" (UniqueName: \"kubernetes.io/projected/44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8-kube-api-access-mgvtb\") pod \"44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8\" (UID: \"44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8\") "
Sep 17 18:23:14.322484 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.322444    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8-kube-api-access-mgvtb" (OuterVolumeSpecName: "kube-api-access-mgvtb") pod "44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8" (UID: "44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8"). InnerVolumeSpecName "kube-api-access-mgvtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:14.421353 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.421316    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgvtb\" (UniqueName: \"kubernetes.io/projected/44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8-kube-api-access-mgvtb\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:14.569608 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.569571    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7628-cad226"]
Sep 17 18:23:14.571613 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.571587    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7628-cad226"]
Sep 17 18:23:14.684444 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.684369    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="370aeb40-2a31-4f94-93a8-7a9cfb70e0b8" path="/var/lib/kubelet/pods/370aeb40-2a31-4f94-93a8-7a9cfb70e0b8/volumes"
Sep 17 18:23:14.684696 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:14.684683    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8" path="/var/lib/kubelet/pods/44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8/volumes"
Sep 17 18:23:15.008124 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.008091    2589 generic.go:356] "Generic (PLEG): container finished" podID="44c77a60-1f99-4eb7-b7ee-c928b2a8e6f8" containerID="aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00" exitCode=137
Sep 17 18:23:15.008492 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.008135    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-edab31"
Sep 17 18:23:15.008492 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.008190    2589 scope.go:117] "RemoveContainer" containerID="aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00"
Sep 17 18:23:15.008492 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:15.008432    2589 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7628-cad226" container="test-curl-7628-cad226"
Sep 17 18:23:15.016988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.016968    2589 scope.go:117] "RemoveContainer" containerID="aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00"
Sep 17 18:23:15.017267 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:15.017246    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00\": container with ID starting with aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00 not found: ID does not exist" containerID="aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00"
Sep 17 18:23:15.017347 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.017277    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00"} err="failed to get container status \"aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00\": rpc error: code = NotFound desc = could not find container \"aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00\": container with ID starting with aa8641e73b738a88fa739841a61f0cdac911890fcd513413858bdbaa4eb63b00 not found: ID does not exist"
Sep 17 18:23:15.031851 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.031824    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:15.139218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.139168    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:15.409939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.409851    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"]
Sep 17 18:23:15.410283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.410266    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="877c7515-e3fe-4cb8-997a-09ba340625a0" containerName="test-curl-7628-cad226"
Sep 17 18:23:15.410386 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.410286    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="877c7515-e3fe-4cb8-997a-09ba340625a0" containerName="test-curl-7628-cad226"
Sep 17 18:23:15.410445 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.410384    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="877c7515-e3fe-4cb8-997a-09ba340625a0" containerName="test-curl-7628-cad226"
Sep 17 18:23:15.412992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.412970    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.415328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.415308    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-d245d137-openshift-default-dockercfg-g4wxp\""
Sep 17 18:23:15.426058 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.426033    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"]
Sep 17 18:23:15.531682 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.531646    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-socket\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.531682 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.531679    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-credential-socket\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.531885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.531699    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istiod-ca-cert\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.531885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.531717    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dfs9\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-kube-api-access-4dfs9\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.531885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.531804    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-certs\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.531885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.531823    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-data\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.532023 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.531889    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-podinfo\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.532023 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.531982    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-token\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.532094 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.532042    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-envoy\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.610020 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.609983    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:15.633304 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633276    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-token\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.633435 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633315    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-envoy\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.633435 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633367    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-socket\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.633435 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633393    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-credential-socket\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.633435 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633415    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istiod-ca-cert\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.633614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633441    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dfs9\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-kube-api-access-4dfs9\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.633614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633482    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-certs\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.633614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633505    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-data\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.633614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633547    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-podinfo\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.633920 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633882    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-credential-socket\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.634044 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633941    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-certs\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.634044 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.633973    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-socket\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.634044 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.634027    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-data\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.634237 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.634209    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istiod-ca-cert\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.636047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.636028    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-envoy\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.636198 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.636182    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-podinfo\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.641153 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.641130    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-token\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.641241 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.641178    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dfs9\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-kube-api-access-4dfs9\") pod \"e2e-mt-d245d137-openshift-default-c948444cb-ggjfw\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") " pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.706437 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.706345    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:15.727587 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.727559    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:15.899276 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:15.899244    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:16.008050 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.007955    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"]
Sep 17 18:23:16.011559 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:23:16.011532    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b1304e_a4b4_4d8e_b186_b6c0623e6953.slice/crio-50ace6c56747cc51de438b6b1d60ff278a72ee357da29a9badf65df2a91906f6 WatchSource:0}: Error finding container 50ace6c56747cc51de438b6b1d60ff278a72ee357da29a9badf65df2a91906f6: Status 404 returned error can't find the container with id 50ace6c56747cc51de438b6b1d60ff278a72ee357da29a9badf65df2a91906f6
Sep 17 18:23:16.014470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.014427    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:23:16.014605 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.014515    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:23:16.014605 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.014564    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:23:16.015318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.015281    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7628-cad226" podUID="877c7515-e3fe-4cb8-997a-09ba340625a0" containerName="test-curl-7628-cad226" containerID="cri-o://21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58" gracePeriod=2
Sep 17 18:23:16.103784 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.103755    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-e13389"
Sep 17 18:23:16.240176 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.240141    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rkbtt\" (UniqueName: \"kubernetes.io/projected/f0a443cb-b33f-4121-a880-02b1a5939e8b-kube-api-access-rkbtt\") pod \"f0a443cb-b33f-4121-a880-02b1a5939e8b\" (UID: \"f0a443cb-b33f-4121-a880-02b1a5939e8b\") "
Sep 17 18:23:16.242497 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.242449    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0a443cb-b33f-4121-a880-02b1a5939e8b-kube-api-access-rkbtt" (OuterVolumeSpecName: "kube-api-access-rkbtt") pod "f0a443cb-b33f-4121-a880-02b1a5939e8b" (UID: "f0a443cb-b33f-4121-a880-02b1a5939e8b"). InnerVolumeSpecName "kube-api-access-rkbtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:16.340944 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.340866    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkbtt\" (UniqueName: \"kubernetes.io/projected/f0a443cb-b33f-4121-a880-02b1a5939e8b-kube-api-access-rkbtt\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:16.596758 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.596672    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:16.683811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:16.683783    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0a443cb-b33f-4121-a880-02b1a5939e8b" path="/var/lib/kubelet/pods/f0a443cb-b33f-4121-a880-02b1a5939e8b/volumes"
Sep 17 18:23:17.021827 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.021785    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" event={"ID":"22b1304e-a4b4-4d8e-b186-b6c0623e6953","Type":"ContainerStarted","Data":"faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455"}
Sep 17 18:23:17.022300 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.021850    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" event={"ID":"22b1304e-a4b4-4d8e-b186-b6c0623e6953","Type":"ContainerStarted","Data":"50ace6c56747cc51de438b6b1d60ff278a72ee357da29a9badf65df2a91906f6"}
Sep 17 18:23:17.022912 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.022888    2589 generic.go:356] "Generic (PLEG): container finished" podID="f0a443cb-b33f-4121-a880-02b1a5939e8b" containerID="43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d" exitCode=137
Sep 17 18:23:17.023042 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.022921    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-e13389"
Sep 17 18:23:17.023042 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.022985    2589 scope.go:117] "RemoveContainer" containerID="43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d"
Sep 17 18:23:17.031429 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.031413    2589 scope.go:117] "RemoveContainer" containerID="43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d"
Sep 17 18:23:17.031686 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:17.031671    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d\": container with ID starting with 43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d not found: ID does not exist" containerID="43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d"
Sep 17 18:23:17.031748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.031694    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d"} err="failed to get container status \"43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d\": rpc error: code = NotFound desc = could not find container \"43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d\": container with ID starting with 43051353a9d57697d0ffab21d38f9546620a97bc0f1cf39392b055f4366c1b8d not found: ID does not exist"
Sep 17 18:23:17.050902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.050861    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" podStartSLOduration=2.050848138 podStartE2EDuration="2.050848138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:17.048352738 +0000 UTC m=+1232.728038592" watchObservedRunningTime="2026-09-17 18:23:17.050848138 +0000 UTC m=+1232.730533992"
Sep 17 18:23:17.432162 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.432051    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:17.728300 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.728210    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:17.733471 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:17.733447    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:18.029833 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:18.029792    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:18.030843 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:18.030826    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:18.201960 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:18.201936    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-cad226"
Sep 17 18:23:18.258292 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:18.258266    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bs7c\" (UniqueName: \"kubernetes.io/projected/877c7515-e3fe-4cb8-997a-09ba340625a0-kube-api-access-8bs7c\") pod \"877c7515-e3fe-4cb8-997a-09ba340625a0\" (UID: \"877c7515-e3fe-4cb8-997a-09ba340625a0\") "
Sep 17 18:23:18.260596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:18.260573    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/877c7515-e3fe-4cb8-997a-09ba340625a0-kube-api-access-8bs7c" (OuterVolumeSpecName: "kube-api-access-8bs7c") pod "877c7515-e3fe-4cb8-997a-09ba340625a0" (UID: "877c7515-e3fe-4cb8-997a-09ba340625a0"). InnerVolumeSpecName "kube-api-access-8bs7c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:18.359238 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:18.359174    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bs7c\" (UniqueName: \"kubernetes.io/projected/877c7515-e3fe-4cb8-997a-09ba340625a0-kube-api-access-8bs7c\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:18.391247 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:18.391206    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:18.683409 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:18.683326    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="877c7515-e3fe-4cb8-997a-09ba340625a0" path="/var/lib/kubelet/pods/877c7515-e3fe-4cb8-997a-09ba340625a0/volumes"
Sep 17 18:23:18.924501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:18.924471    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:19.015397 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:19.015373    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:19.035507 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:19.035483    2589 generic.go:356] "Generic (PLEG): container finished" podID="877c7515-e3fe-4cb8-997a-09ba340625a0" containerID="21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58" exitCode=137
Sep 17 18:23:19.035861 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:19.035525    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7628-cad226"
Sep 17 18:23:19.035861 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:19.035578    2589 scope.go:117] "RemoveContainer" containerID="21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58"
Sep 17 18:23:19.044111 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:19.044094    2589 scope.go:117] "RemoveContainer" containerID="21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58"
Sep 17 18:23:19.044372 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:19.044352    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58\": container with ID starting with 21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58 not found: ID does not exist" containerID="21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58"
Sep 17 18:23:19.044435 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:19.044382    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58"} err="failed to get container status \"21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58\": rpc error: code = NotFound desc = could not find container \"21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58\": container with ID starting with 21c7571f1d0632f7f1392fe2ed0bac8895430d11fb7288730d4cda0c95042a58 not found: ID does not exist"
Sep 17 18:23:19.241934 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:19.241809    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:19.810564 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:19.810524    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:19.910537 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:19.910501    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:20.083748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.083657    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:20.386611 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.386520    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"]
Sep 17 18:23:20.389796 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.389773    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"
Sep 17 18:23:20.392309 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.392287    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:23:20.392430 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.392293    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:23:20.399278 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.399252    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"]
Sep 17 18:23:20.477013 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.476986    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c36257-ea3b-45b0-bb7e-16fd467ac77e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw\" (UID: \"46c36257-ea3b-45b0-bb7e-16fd467ac77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"
Sep 17 18:23:20.578313 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.578279    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c36257-ea3b-45b0-bb7e-16fd467ac77e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw\" (UID: \"46c36257-ea3b-45b0-bb7e-16fd467ac77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"
Sep 17 18:23:20.578891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.578871    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c36257-ea3b-45b0-bb7e-16fd467ac77e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw\" (UID: \"46c36257-ea3b-45b0-bb7e-16fd467ac77e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"
Sep 17 18:23:20.701702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.701632    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"
Sep 17 18:23:20.970987 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:20.970925    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"]
Sep 17 18:23:20.973416 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:23:20.973389    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46c36257_ea3b_45b0_bb7e_16fd467ac77e.slice/crio-2614f61e64ed592695b6d0df7c1a04449907a494c19067eb988dc5e565d8f745 WatchSource:0}: Error finding container 2614f61e64ed592695b6d0df7c1a04449907a494c19067eb988dc5e565d8f745: Status 404 returned error can't find the container with id 2614f61e64ed592695b6d0df7c1a04449907a494c19067eb988dc5e565d8f745
Sep 17 18:23:21.049074 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:21.049043    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw" event={"ID":"46c36257-ea3b-45b0-bb7e-16fd467ac77e","Type":"ContainerStarted","Data":"2614f61e64ed592695b6d0df7c1a04449907a494c19067eb988dc5e565d8f745"}
Sep 17 18:23:21.051367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:21.051330    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:21.881572 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:21.881530    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:22.055434 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:22.055399    2589 generic.go:356] "Generic (PLEG): container finished" podID="46c36257-ea3b-45b0-bb7e-16fd467ac77e" containerID="35629e918ad48436ab6ed4fbfae6b7516e6c5a81f51b328d708053cf9e3fc3f0" exitCode=0
Sep 17 18:23:22.055612 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:22.055463    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw" event={"ID":"46c36257-ea3b-45b0-bb7e-16fd467ac77e","Type":"ContainerDied","Data":"35629e918ad48436ab6ed4fbfae6b7516e6c5a81f51b328d708053cf9e3fc3f0"}
Sep 17 18:23:22.736576 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:22.736534    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:23.187811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:23.187787    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"
Sep 17 18:23:23.201875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:23.201850    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c36257-ea3b-45b0-bb7e-16fd467ac77e-maas-api-service-ca\") pod \"46c36257-ea3b-45b0-bb7e-16fd467ac77e\" (UID: \"46c36257-ea3b-45b0-bb7e-16fd467ac77e\") "
Sep 17 18:23:23.202168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:23.202147    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46c36257-ea3b-45b0-bb7e-16fd467ac77e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "46c36257-ea3b-45b0-bb7e-16fd467ac77e" (UID: "46c36257-ea3b-45b0-bb7e-16fd467ac77e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:23.303466 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:23.303423    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46c36257-ea3b-45b0-bb7e-16fd467ac77e-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:23.500548 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:23.500463    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:23.568330 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:23.568275    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:23.706484 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:23.706448    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:24.023482 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.023456    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:24.066049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.065998    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw" event={"ID":"46c36257-ea3b-45b0-bb7e-16fd467ac77e","Type":"ContainerDied","Data":"2614f61e64ed592695b6d0df7c1a04449907a494c19067eb988dc5e565d8f745"}
Sep 17 18:23:24.066049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.065999    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"
Sep 17 18:23:24.066244 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.066063    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2614f61e64ed592695b6d0df7c1a04449907a494c19067eb988dc5e565d8f745"
Sep 17 18:23:24.108692 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.108657    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:24.157513 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.157485    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"]
Sep 17 18:23:24.157848 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.157835    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46c36257-ea3b-45b0-bb7e-16fd467ac77e" containerName="revoke-keys"
Sep 17 18:23:24.157892 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.157851    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="46c36257-ea3b-45b0-bb7e-16fd467ac77e" containerName="revoke-keys"
Sep 17 18:23:24.157932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.157921    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="46c36257-ea3b-45b0-bb7e-16fd467ac77e" containerName="revoke-keys"
Sep 17 18:23:24.160279 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.160262    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:24.163039 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.163020    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-d245d137\""
Sep 17 18:23:24.163177 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.163055    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-d245d137-dockercfg-6mhb2\""
Sep 17 18:23:24.170629 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.170606    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"]
Sep 17 18:23:24.209341 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.209319    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"]
Sep 17 18:23:24.211848 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.211832    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:24.212165 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.212145    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173e2c4-c0a1-4650-9870-d61191a4078b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2\" (UID: \"f173e2c4-c0a1-4650-9870-d61191a4078b\") " pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:24.212235 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.212179    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68mzk\" (UniqueName: \"kubernetes.io/projected/f173e2c4-c0a1-4650-9870-d61191a4078b-kube-api-access-68mzk\") pod \"payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2\" (UID: \"f173e2c4-c0a1-4650-9870-d61191a4078b\") " pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:24.223885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.223864    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"]
Sep 17 18:23:24.312942 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.312857    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173e2c4-c0a1-4650-9870-d61191a4078b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2\" (UID: \"f173e2c4-c0a1-4650-9870-d61191a4078b\") " pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:24.312942 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.312898    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68mzk\" (UniqueName: \"kubernetes.io/projected/f173e2c4-c0a1-4650-9870-d61191a4078b-kube-api-access-68mzk\") pod \"payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2\" (UID: \"f173e2c4-c0a1-4650-9870-d61191a4078b\") " pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:24.312942 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.312927    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwmdd\" (UniqueName: \"kubernetes.io/projected/0b15b532-82ca-4f8f-a5b7-9695cc719737-kube-api-access-mwmdd\") pod \"payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq\" (UID: \"0b15b532-82ca-4f8f-a5b7-9695cc719737\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:24.313254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.313015    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b532-82ca-4f8f-a5b7-9695cc719737-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq\" (UID: \"0b15b532-82ca-4f8f-a5b7-9695cc719737\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:24.313664 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.313641    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173e2c4-c0a1-4650-9870-d61191a4078b-plugins-config-volume\") pod \"payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2\" (UID: \"f173e2c4-c0a1-4650-9870-d61191a4078b\") " pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:24.321737 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.321704    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68mzk\" (UniqueName: \"kubernetes.io/projected/f173e2c4-c0a1-4650-9870-d61191a4078b-kube-api-access-68mzk\") pod \"payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2\" (UID: \"f173e2c4-c0a1-4650-9870-d61191a4078b\") " pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:24.414246 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.414215    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b532-82ca-4f8f-a5b7-9695cc719737-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq\" (UID: \"0b15b532-82ca-4f8f-a5b7-9695cc719737\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:24.414400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.414307    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwmdd\" (UniqueName: \"kubernetes.io/projected/0b15b532-82ca-4f8f-a5b7-9695cc719737-kube-api-access-mwmdd\") pod \"payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq\" (UID: \"0b15b532-82ca-4f8f-a5b7-9695cc719737\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:24.414908 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.414889    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b532-82ca-4f8f-a5b7-9695cc719737-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq\" (UID: \"0b15b532-82ca-4f8f-a5b7-9695cc719737\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:24.423022 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.422974    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwmdd\" (UniqueName: \"kubernetes.io/projected/0b15b532-82ca-4f8f-a5b7-9695cc719737-kube-api-access-mwmdd\") pod \"payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq\" (UID: \"0b15b532-82ca-4f8f-a5b7-9695cc719737\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:24.470992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.470964    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:24.521894 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.521863    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:24.648242 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.648210    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:24.745351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.745325    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"]
Sep 17 18:23:24.747195 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:23:24.747171    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf173e2c4_c0a1_4650_9870_d61191a4078b.slice/crio-3dd488a8aaa5c8b9a428b7b98cfe297bdecd99fb58367193cef7496c828c9fd8 WatchSource:0}: Error finding container 3dd488a8aaa5c8b9a428b7b98cfe297bdecd99fb58367193cef7496c828c9fd8: Status 404 returned error can't find the container with id 3dd488a8aaa5c8b9a428b7b98cfe297bdecd99fb58367193cef7496c828c9fd8
Sep 17 18:23:24.798255 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:24.798227    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"]
Sep 17 18:23:24.799622 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:23:24.799582    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b15b532_82ca_4f8f_a5b7_9695cc719737.slice/crio-5e30aaddbd5ae2aaaae3e1dc60954c13e902e58dfba0d916915a5691e700feff WatchSource:0}: Error finding container 5e30aaddbd5ae2aaaae3e1dc60954c13e902e58dfba0d916915a5691e700feff: Status 404 returned error can't find the container with id 5e30aaddbd5ae2aaaae3e1dc60954c13e902e58dfba0d916915a5691e700feff
Sep 17 18:23:25.074180 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:25.074131    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq" event={"ID":"0b15b532-82ca-4f8f-a5b7-9695cc719737","Type":"ContainerStarted","Data":"c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853"}
Sep 17 18:23:25.074363 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:25.074201    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq" event={"ID":"0b15b532-82ca-4f8f-a5b7-9695cc719737","Type":"ContainerStarted","Data":"5e30aaddbd5ae2aaaae3e1dc60954c13e902e58dfba0d916915a5691e700feff"}
Sep 17 18:23:25.074363 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:25.074247    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:25.075782 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:25.075748    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2" event={"ID":"f173e2c4-c0a1-4650-9870-d61191a4078b","Type":"ContainerStarted","Data":"a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365"}
Sep 17 18:23:25.075888 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:25.075792    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:25.075888 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:25.075811    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2" event={"ID":"f173e2c4-c0a1-4650-9870-d61191a4078b","Type":"ContainerStarted","Data":"3dd488a8aaa5c8b9a428b7b98cfe297bdecd99fb58367193cef7496c828c9fd8"}
Sep 17 18:23:25.093407 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:25.093328    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq" podStartSLOduration=1.093315591 podStartE2EDuration="1.093315591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:25.090471643 +0000 UTC m=+1240.770157504" watchObservedRunningTime="2026-09-17 18:23:25.093315591 +0000 UTC m=+1240.773001445"
Sep 17 18:23:25.107360 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:25.107315    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2" podStartSLOduration=1.107299698 podStartE2EDuration="1.107299698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:23:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:23:25.1052722 +0000 UTC m=+1240.784958055" watchObservedRunningTime="2026-09-17 18:23:25.107299698 +0000 UTC m=+1240.786985552"
Sep 17 18:23:25.456075 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:25.455972    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:26.140215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:26.140183    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:27.083860 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:27.083831    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:28.296612 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:28.296582    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:28.803141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:28.803107    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:30.520304 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:30.520268    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:31.126777 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.126744    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:31.149893 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.149867    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"]
Sep 17 18:23:31.150113 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.150083    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd" podUID="78c51caf-eacb-4774-945f-87c518c66e8d" containerName="payload-processing" containerID="cri-o://203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5" gracePeriod=30
Sep 17 18:23:31.193134 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.193093    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:31.250879 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.250848    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:31.383228 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.383196    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:31.400642 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.400616    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:23:31.405153 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.405130    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"]
Sep 17 18:23:31.423189 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.423156    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-99ac10-df7jw"]
Sep 17 18:23:31.480095 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.480069    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c51caf-eacb-4774-945f-87c518c66e8d-plugins-config-volume\") pod \"78c51caf-eacb-4774-945f-87c518c66e8d\" (UID: \"78c51caf-eacb-4774-945f-87c518c66e8d\") "
Sep 17 18:23:31.480371 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.480335    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62mxg\" (UniqueName: \"kubernetes.io/projected/78c51caf-eacb-4774-945f-87c518c66e8d-kube-api-access-62mxg\") pod \"78c51caf-eacb-4774-945f-87c518c66e8d\" (UID: \"78c51caf-eacb-4774-945f-87c518c66e8d\") "
Sep 17 18:23:31.480631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.480331    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78c51caf-eacb-4774-945f-87c518c66e8d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "78c51caf-eacb-4774-945f-87c518c66e8d" (UID: "78c51caf-eacb-4774-945f-87c518c66e8d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:23:31.482377 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.482339    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:31.483110 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.483088    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78c51caf-eacb-4774-945f-87c518c66e8d-kube-api-access-62mxg" (OuterVolumeSpecName: "kube-api-access-62mxg") pod "78c51caf-eacb-4774-945f-87c518c66e8d" (UID: "78c51caf-eacb-4774-945f-87c518c66e8d"). InnerVolumeSpecName "kube-api-access-62mxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:23:31.581180 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.581154    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78c51caf-eacb-4774-945f-87c518c66e8d-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:31.581180 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.581177    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62mxg\" (UniqueName: \"kubernetes.io/projected/78c51caf-eacb-4774-945f-87c518c66e8d-kube-api-access-62mxg\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:23:31.738637 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.738549    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:31.848500 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:31.848466    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:32.113575 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.113545    2589 generic.go:356] "Generic (PLEG): container finished" podID="78c51caf-eacb-4774-945f-87c518c66e8d" containerID="203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5" exitCode=0
Sep 17 18:23:32.113762 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.113586    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"
Sep 17 18:23:32.113762 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.113630    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd" event={"ID":"78c51caf-eacb-4774-945f-87c518c66e8d","Type":"ContainerDied","Data":"203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5"}
Sep 17 18:23:32.113762 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.113685    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd" event={"ID":"78c51caf-eacb-4774-945f-87c518c66e8d","Type":"ContainerDied","Data":"4597200aa1faef76684cf892e38b253e0e85a039dc03622114809e8c5318bffe"}
Sep 17 18:23:32.113762 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.113709    2589 scope.go:117] "RemoveContainer" containerID="203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5"
Sep 17 18:23:32.123370 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.123353    2589 scope.go:117] "RemoveContainer" containerID="203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5"
Sep 17 18:23:32.123624 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:23:32.123602    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5\": container with ID starting with 203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5 not found: ID does not exist" containerID="203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5"
Sep 17 18:23:32.123689 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.123632    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5"} err="failed to get container status \"203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5\": rpc error: code = NotFound desc = could not find container \"203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5\": container with ID starting with 203b046b6ce2bd52ffb0d70df230620258ddfefaa83676ec967743b231aaece5 not found: ID does not exist"
Sep 17 18:23:32.135769 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.135745    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"]
Sep 17 18:23:32.140280 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.140259    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-99ac10-76ccc4dc57-ggxcd"]
Sep 17 18:23:32.566298 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.566264    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:32.683500 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.683471    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46c36257-ea3b-45b0-bb7e-16fd467ac77e" path="/var/lib/kubelet/pods/46c36257-ea3b-45b0-bb7e-16fd467ac77e/volumes"
Sep 17 18:23:32.683870 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:32.683828    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78c51caf-eacb-4774-945f-87c518c66e8d" path="/var/lib/kubelet/pods/78c51caf-eacb-4774-945f-87c518c66e8d/volumes"
Sep 17 18:23:33.433447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:33.433411    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:34.256911 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:34.256881    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:35.016397 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:35.016363    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:35.727766 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:35.727723    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:36.084186 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:36.084137    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:23:36.084354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:36.084319    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:23:36.819900 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:36.819868    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:37.154414 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:37.154319    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:38.061436 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:38.061395    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:38.732039 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:38.731984    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:38.830299 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:38.830267    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:39.200617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:39.200582    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:23:39.667838 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:39.667794    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:40.907391 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:40.907333    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:41.269396 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:41.269363    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:48.787559 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:48.787519    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:23:49.071250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:49.071153    2589 patch_prober.go:28] interesting pod/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 17 18:23:49.071250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:49.071221    2589 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 17 18:23:50.220473 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:50.220438    2589 generic.go:356] "Generic (PLEG): container finished" podID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerID="faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455" exitCode=0
Sep 17 18:23:50.220877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:50.220517    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" event={"ID":"22b1304e-a4b4-4d8e-b186-b6c0623e6953","Type":"ContainerDied","Data":"faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455"}
Sep 17 18:23:50.220994 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:50.220977    2589 scope.go:117] "RemoveContainer" containerID="faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455"
Sep 17 18:23:50.227941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:50.222391    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:23:50.227941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:50.222571    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:23:50.227941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:50.222679    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:23:51.227216 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:51.227179    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" event={"ID":"22b1304e-a4b4-4d8e-b186-b6c0623e6953","Type":"ContainerStarted","Data":"ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88"}
Sep 17 18:23:51.727942 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:51.727901    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:51.728217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:51.728193    2589 patch_prober.go:28] interesting pod/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 18:23:51.728307 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:51.728249    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 18:23:52.728659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:52.728621    2589 patch_prober.go:28] interesting pod/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 18:23:52.729094 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:52.728693    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 18:23:53.728625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:53.728586    2589 patch_prober.go:28] interesting pod/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 18:23:53.728813 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:53.728645    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 18:23:54.728789 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:54.728753    2589 patch_prober.go:28] interesting pod/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 18:23:54.728968 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:54.728811    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 18:23:55.728026 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:55.727967    2589 patch_prober.go:28] interesting pod/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body=
Sep 17 18:23:55.728231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:55.728037    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused"
Sep 17 18:23:56.731547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:56.731513    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:56.731956 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:56.731795    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:56.732540 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:56.732521    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:23:59.393383 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:23:59.393350    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:13.117650 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:13.117613    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:24:14.412271 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:14.412224    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:14.640120 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:14.640079    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:15.225771 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.225735    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:15.520722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.520694    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"]
Sep 17 18:24:15.521120 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.521062    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78c51caf-eacb-4774-945f-87c518c66e8d" containerName="payload-processing"
Sep 17 18:24:15.521120 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.521074    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="78c51caf-eacb-4774-945f-87c518c66e8d" containerName="payload-processing"
Sep 17 18:24:15.521206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.521139    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="78c51caf-eacb-4774-945f-87c518c66e8d" containerName="payload-processing"
Sep 17 18:24:15.522975 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.522957    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.526167 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.526148    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-95a92b-openshift-default-dockercfg-vqhhd\""
Sep 17 18:24:15.540150 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.540127    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"]
Sep 17 18:24:15.546681 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.546656    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:15.598029 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.597978    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-token\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.598178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.598063    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-data\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.598178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.598084    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-credential-socket\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.598178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.598115    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-socket\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.598288 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.598184    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzmb8\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-kube-api-access-tzmb8\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.598288 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.598208    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-podinfo\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.598288 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.598234    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-certs\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.598288 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.598254    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-envoy\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.598288 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.598281    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761b7aa1-f9cb-476a-8943-764e98cb31ac-istiod-ca-cert\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.698775 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.698739    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-data\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.698775 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.698775    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-credential-socket\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.698798    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-socket\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.698844    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzmb8\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-kube-api-access-tzmb8\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.698868    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-podinfo\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.698898    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-certs\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.698930    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-envoy\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.698983    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761b7aa1-f9cb-476a-8943-764e98cb31ac-istiod-ca-cert\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699047 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.699045    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-token\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699421 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.699246    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-data\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699421 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.699265    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-credential-socket\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699421 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.699376    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-certs\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699525 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.699513    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-socket\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.699800 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.699781    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761b7aa1-f9cb-476a-8943-764e98cb31ac-istiod-ca-cert\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.701457 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.701433    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-envoy\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.701615 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.701594    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-podinfo\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.707950 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.707924    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-token\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.708293 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.708277    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzmb8\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-kube-api-access-tzmb8\") pod \"e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") " pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:15.820920 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.820842    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:15.832776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:15.832757    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:16.114226 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:16.114203    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"]
Sep 17 18:24:16.125041 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:16.124970    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:24:16.125165 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:16.125081    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:24:16.125165 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:16.125113    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:24:16.357748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:16.357702    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" event={"ID":"761b7aa1-f9cb-476a-8943-764e98cb31ac","Type":"ContainerStarted","Data":"2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979"}
Sep 17 18:24:16.357927 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:16.357808    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" event={"ID":"761b7aa1-f9cb-476a-8943-764e98cb31ac","Type":"ContainerStarted","Data":"b9335243997c06230c7431e862a7077565ef50404520b8cb90b19e2ff8d4eb1d"}
Sep 17 18:24:16.383243 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:16.383126    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" podStartSLOduration=1.383104879 podStartE2EDuration="1.383104879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:16.380330338 +0000 UTC m=+1292.060016194" watchObservedRunningTime="2026-09-17 18:24:16.383104879 +0000 UTC m=+1292.062790735"
Sep 17 18:24:16.833503 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:16.833473    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:17.553551 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:17.553517    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:17.838559 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:17.838478    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:18.366646 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:18.366618    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:18.367483 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:18.367462    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:18.437029 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:18.436976    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:18.519242 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:18.519206    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:19.360407 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:19.360366    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:20.355523 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:20.355490    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:21.112313 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:21.112279    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:21.921938 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:21.921904    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:22.762846 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:22.762806    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:22.863449 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:22.863414    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:22.962902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:22.962855    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:23.287778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:23.287742    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:23.386124 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:23.386088    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:36.008744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:36.008707    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:24:41.680311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.680271    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"]
Sep 17 18:24:41.693298 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.693271    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.694905 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.694869    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"]
Sep 17 18:24:41.696405 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.696383    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d245d137\"/\"default-dockercfg-lggps\""
Sep 17 18:24:41.697175 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.697150    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d245d137\"/\"kube-root-ca.crt\""
Sep 17 18:24:41.697407 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.697385    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-d245d137\"/\"e2e-lifecycle-model-d245d137-kserve-self-signed-certs\""
Sep 17 18:24:41.697676 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.697619    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-d245d137\"/\"openshift-service-ca.crt\""
Sep 17 18:24:41.737868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.737840    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-home\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.738028 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.737879    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a57a31-0ed7-419d-b59d-435bf124d15a-tls-certs\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.738028 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.737899    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pvhr\" (UniqueName: \"kubernetes.io/projected/b7a57a31-0ed7-419d-b59d-435bf124d15a-kube-api-access-6pvhr\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.738028 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.737981    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-model-cache\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.738211 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.738083    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-dshm\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.738211 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.738112    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-tmp-dir\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.839333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.839298    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-dshm\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.839333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.839336    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-tmp-dir\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.839590 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.839419    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-home\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.839590 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.839469    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a57a31-0ed7-419d-b59d-435bf124d15a-tls-certs\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.839590 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.839497    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pvhr\" (UniqueName: \"kubernetes.io/projected/b7a57a31-0ed7-419d-b59d-435bf124d15a-kube-api-access-6pvhr\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.839590 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.839547    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-model-cache\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.839865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.839841    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-tmp-dir\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.839925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.839886    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-model-cache\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.839961 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.839936    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-home\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.841748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.841721    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-dshm\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.842044 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.842022    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a57a31-0ed7-419d-b59d-435bf124d15a-tls-certs\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.848032 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.847981    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pvhr\" (UniqueName: \"kubernetes.io/projected/b7a57a31-0ed7-419d-b59d-435bf124d15a-kube-api-access-6pvhr\") pod \"e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") " pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:41.995053 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:41.994947    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:42.009042 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:42.008998    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:42.283963 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:42.282582    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"]
Sep 17 18:24:42.494121 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:42.493837    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps" event={"ID":"b7a57a31-0ed7-419d-b59d-435bf124d15a","Type":"ContainerStarted","Data":"c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698"}
Sep 17 18:24:42.494121 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:42.493890    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps" event={"ID":"b7a57a31-0ed7-419d-b59d-435bf124d15a","Type":"ContainerStarted","Data":"01a39f12952b53979570698151d1cc6ff65f402d0fee1fe7fe3208323625bbe8"}
Sep 17 18:24:42.514581 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:42.514529    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps" podStartSLOduration=1.514515021 podStartE2EDuration="1.514515021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:42.513052992 +0000 UTC m=+1318.192738847" watchObservedRunningTime="2026-09-17 18:24:42.514515021 +0000 UTC m=+1318.194200896"
Sep 17 18:24:44.848520 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:44.848478    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:45.645057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:45.644987    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:46.322911 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:46.322867    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:47.209853 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:47.209818    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:47.523851 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:47.523816    2589 generic.go:356] "Generic (PLEG): container finished" podID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerID="2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979" exitCode=0
Sep 17 18:24:47.524251 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:47.523891    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" event={"ID":"761b7aa1-f9cb-476a-8943-764e98cb31ac","Type":"ContainerDied","Data":"2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979"}
Sep 17 18:24:47.524384 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:47.524367    2589 scope.go:117] "RemoveContainer" containerID="2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979"
Sep 17 18:24:47.525585 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:47.525555    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:24:47.525656 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:47.525644    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:24:47.525694 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:47.525681    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:24:47.928573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:47.928475    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:48.366729 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:48.366696    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:48.373234 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:48.373203    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:48.531392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:48.531358    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" event={"ID":"761b7aa1-f9cb-476a-8943-764e98cb31ac","Type":"ContainerStarted","Data":"be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a"}
Sep 17 18:24:48.622673 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:48.622590    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:48.723918 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:48.723882    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:48.833868 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:48.833836    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:48.835254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:48.835229    2589 patch_prober.go:28] interesting pod/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 18:24:48.835370 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:48.835309    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 18:24:49.010482 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:49.010430    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:49.428889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:49.428800    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:49.529040 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:49.528943    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:49.834052 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:49.834000    2589 patch_prober.go:28] interesting pod/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 18:24:49.834435 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:49.834078    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 18:24:50.833499 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:50.833464    2589 patch_prober.go:28] interesting pod/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 18:24:50.833685 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:50.833532    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 18:24:51.542139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:51.542094    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:51.833563 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:51.833480    2589 patch_prober.go:28] interesting pod/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 18:24:51.833563 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:51.833547    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 18:24:52.009190 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:52.009156    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:52.009451 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:52.009433    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:52.022720 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:52.022690    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:52.206407 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:52.206308    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:52.292936 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:52.292900    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:52.556893 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:52.556850    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:52.563327 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:52.563301    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:24:52.834163 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:52.834083    2589 patch_prober.go:28] interesting pod/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 17 18:24:52.834163 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:52.834149    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 17 18:24:53.429310 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:53.429268    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:53.869383 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:53.869335    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:53.869864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:53.869738    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:53.870992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:53.870969    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:24:54.209406 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:54.209308    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:55.085595 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:55.085562    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:55.753539 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:55.753462    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:56.662784 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:56.662751    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:57.465986 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:57.465933    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:57.525644 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:57.525612    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:57.918957 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:57.918921    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:58.013354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.013319    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:58.045654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.045626    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"]
Sep 17 18:24:58.048138 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.048123    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:24:58.050616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.050600    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d81d7c\""
Sep 17 18:24:58.059122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.059100    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"]
Sep 17 18:24:58.088192 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.088163    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:24:58.103048 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.103024    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3dd22b-a357-4b33-a1a6-996d9121ac25-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l\" (UID: \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:24:58.103143 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.103052    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbmjm\" (UniqueName: \"kubernetes.io/projected/ba3dd22b-a357-4b33-a1a6-996d9121ac25-kube-api-access-tbmjm\") pod \"payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l\" (UID: \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:24:58.205151 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.205062    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3dd22b-a357-4b33-a1a6-996d9121ac25-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l\" (UID: \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:24:58.205151 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.205100    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbmjm\" (UniqueName: \"kubernetes.io/projected/ba3dd22b-a357-4b33-a1a6-996d9121ac25-kube-api-access-tbmjm\") pod \"payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l\" (UID: \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:24:58.205825 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.205801    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3dd22b-a357-4b33-a1a6-996d9121ac25-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l\" (UID: \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:24:58.214152 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.214127    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbmjm\" (UniqueName: \"kubernetes.io/projected/ba3dd22b-a357-4b33-a1a6-996d9121ac25-kube-api-access-tbmjm\") pod \"payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l\" (UID: \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:24:58.359039 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.358995    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:24:58.628030 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:58.627975    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"]
Sep 17 18:24:58.630071 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:24:58.630039    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba3dd22b_a357_4b33_a1a6_996d9121ac25.slice/crio-380b0482af583efc601f03f80f7c3455e0b5d78fb3b145316a336675962b4df9 WatchSource:0}: Error finding container 380b0482af583efc601f03f80f7c3455e0b5d78fb3b145316a336675962b4df9: Status 404 returned error can't find the container with id 380b0482af583efc601f03f80f7c3455e0b5d78fb3b145316a336675962b4df9
Sep 17 18:24:59.586577 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:59.586543    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l" event={"ID":"ba3dd22b-a357-4b33-a1a6-996d9121ac25","Type":"ContainerStarted","Data":"31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03"}
Sep 17 18:24:59.587058 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:59.586596    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l" event={"ID":"ba3dd22b-a357-4b33-a1a6-996d9121ac25","Type":"ContainerStarted","Data":"380b0482af583efc601f03f80f7c3455e0b5d78fb3b145316a336675962b4df9"}
Sep 17 18:24:59.587058 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:59.586628    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:24:59.603413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:24:59.603359    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l" podStartSLOduration=1.603340304 podStartE2EDuration="1.603340304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:24:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:24:59.602971879 +0000 UTC m=+1335.282657733" watchObservedRunningTime="2026-09-17 18:24:59.603340304 +0000 UTC m=+1335.283026160"
Sep 17 18:25:10.596196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:10.596163    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:25:11.202447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:11.202412    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:25:12.726521 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:12.726483    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:19.690695 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:19.690657    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:20.383085 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:20.383040    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:21.240842 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:21.240808    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:21.965069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:21.965033    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:22.237789 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:22.237707    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:22.740946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:22.740907    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:23.635061 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:23.635028    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:24.389175 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:24.389135    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:25.223272 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:25.223235    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:26.532264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:26.532229    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"]
Sep 17 18:25:26.545625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:26.545588    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:32.223158 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:32.223122    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:41.552108 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.552046    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948" podUID="ea6562fa-9154-4fc7-9881-af4a4f9b49da" containerName="main" containerID="cri-o://bbce35ee340da13969245d13280393551e5a24a1e2a9c5c180ecfd0503483656" gracePeriod=45
Sep 17 18:25:41.603106 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.603063    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:41.797329 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.797299    2589 generic.go:356] "Generic (PLEG): container finished" podID="ea6562fa-9154-4fc7-9881-af4a4f9b49da" containerID="bbce35ee340da13969245d13280393551e5a24a1e2a9c5c180ecfd0503483656" exitCode=0
Sep 17 18:25:41.797451 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.797352    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948" event={"ID":"ea6562fa-9154-4fc7-9881-af4a4f9b49da","Type":"ContainerDied","Data":"bbce35ee340da13969245d13280393551e5a24a1e2a9c5c180ecfd0503483656"}
Sep 17 18:25:41.797451 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.797383    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948" event={"ID":"ea6562fa-9154-4fc7-9881-af4a4f9b49da","Type":"ContainerDied","Data":"eb5b9f641a923d03ee20e41ffd321222faed6fe47cefa722d10df9efc8feca10"}
Sep 17 18:25:41.797451 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.797394    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb5b9f641a923d03ee20e41ffd321222faed6fe47cefa722d10df9efc8feca10"
Sep 17 18:25:41.803748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.803693    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:25:41.926906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.926867    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tmp-dir\") pod \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") "
Sep 17 18:25:41.926906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.926910    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-home\") pod \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") "
Sep 17 18:25:41.927156 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.926932    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tls-certs\") pod \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") "
Sep 17 18:25:41.927156 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.926970    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rj85z\" (UniqueName: \"kubernetes.io/projected/ea6562fa-9154-4fc7-9881-af4a4f9b49da-kube-api-access-rj85z\") pod \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") "
Sep 17 18:25:41.927156 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.927080    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-dshm\") pod \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") "
Sep 17 18:25:41.927156 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.927113    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-model-cache\") pod \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\" (UID: \"ea6562fa-9154-4fc7-9881-af4a4f9b49da\") "
Sep 17 18:25:41.927432 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.927220    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-home" (OuterVolumeSpecName: "home") pod "ea6562fa-9154-4fc7-9881-af4a4f9b49da" (UID: "ea6562fa-9154-4fc7-9881-af4a4f9b49da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:41.927432 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.927225    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea6562fa-9154-4fc7-9881-af4a4f9b49da" (UID: "ea6562fa-9154-4fc7-9881-af4a4f9b49da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:41.927432 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.927414    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea6562fa-9154-4fc7-9881-af4a4f9b49da" (UID: "ea6562fa-9154-4fc7-9881-af4a4f9b49da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:41.927561 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.927472    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:25:41.927561 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.927491    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:25:41.929457 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.929429    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-dshm" (OuterVolumeSpecName: "dshm") pod "ea6562fa-9154-4fc7-9881-af4a4f9b49da" (UID: "ea6562fa-9154-4fc7-9881-af4a4f9b49da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:25:41.929560 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.929455    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea6562fa-9154-4fc7-9881-af4a4f9b49da" (UID: "ea6562fa-9154-4fc7-9881-af4a4f9b49da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:25:41.929560 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:41.929486    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea6562fa-9154-4fc7-9881-af4a4f9b49da-kube-api-access-rj85z" (OuterVolumeSpecName: "kube-api-access-rj85z") pod "ea6562fa-9154-4fc7-9881-af4a4f9b49da" (UID: "ea6562fa-9154-4fc7-9881-af4a4f9b49da"). InnerVolumeSpecName "kube-api-access-rj85z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:25:42.028768 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:42.028734    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rj85z\" (UniqueName: \"kubernetes.io/projected/ea6562fa-9154-4fc7-9881-af4a4f9b49da-kube-api-access-rj85z\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:25:42.028768 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:42.028762    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:25:42.028768 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:42.028774    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea6562fa-9154-4fc7-9881-af4a4f9b49da-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:25:42.028988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:42.028783    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea6562fa-9154-4fc7-9881-af4a4f9b49da-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:25:42.803030 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:42.802971    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"
Sep 17 18:25:42.819078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:42.819052    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"]
Sep 17 18:25:42.824969 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:42.824942    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/e2e-conflict-a-92400e60-kserve-7fcbbb57c9-xq948"]
Sep 17 18:25:44.684090 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:44.684056    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea6562fa-9154-4fc7-9881-af4a4f9b49da" path="/var/lib/kubelet/pods/ea6562fa-9154-4fc7-9881-af4a4f9b49da/volumes"
Sep 17 18:25:44.904752 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:44.904710    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:25:46.752535 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:46.752504    2589 scope.go:117] "RemoveContainer" containerID="4b5a2479c45231fe78ef8485deb252a31d4e4ac0eb898f4a03b2c8668d56b371"
Sep 17 18:25:50.892631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:50.892598    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"]
Sep 17 18:25:50.893024 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:50.892962    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea6562fa-9154-4fc7-9881-af4a4f9b49da" containerName="main"
Sep 17 18:25:50.893024 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:50.892973    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea6562fa-9154-4fc7-9881-af4a4f9b49da" containerName="main"
Sep 17 18:25:50.893133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:50.893063    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea6562fa-9154-4fc7-9881-af4a4f9b49da" containerName="main"
Sep 17 18:25:50.895859 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:50.895843    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"
Sep 17 18:25:50.898139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:50.898108    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:25:50.898139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:50.898139    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:25:50.904658 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:50.904632    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"]
Sep 17 18:25:51.013327 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.013282    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee09db-7efe-4b6a-8962-1c455bd9998c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d245d137-rt5ts\" (UID: \"d2ee09db-7efe-4b6a-8962-1c455bd9998c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"
Sep 17 18:25:51.114163 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.114131    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee09db-7efe-4b6a-8962-1c455bd9998c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d245d137-rt5ts\" (UID: \"d2ee09db-7efe-4b6a-8962-1c455bd9998c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"
Sep 17 18:25:51.114770 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.114751    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee09db-7efe-4b6a-8962-1c455bd9998c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-d245d137-rt5ts\" (UID: \"d2ee09db-7efe-4b6a-8962-1c455bd9998c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"
Sep 17 18:25:51.206544 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.206455    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"
Sep 17 18:25:51.237578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.237546    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:51.478736 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.478711    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"]
Sep 17 18:25:51.481135 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:25:51.481106    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2ee09db_7efe_4b6a_8962_1c455bd9998c.slice/crio-7f582910ae060ce47d322e7623fa1262dbf6ce92b371767a579523487aee76c1 WatchSource:0}: Error finding container 7f582910ae060ce47d322e7623fa1262dbf6ce92b371767a579523487aee76c1: Status 404 returned error can't find the container with id 7f582910ae060ce47d322e7623fa1262dbf6ce92b371767a579523487aee76c1
Sep 17 18:25:51.848325 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.848294    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts" event={"ID":"d2ee09db-7efe-4b6a-8962-1c455bd9998c","Type":"ContainerStarted","Data":"fe54ef2e17ffaa25bd35681c6d32faad355a846aed990644baf2919fe74c4010"}
Sep 17 18:25:51.848521 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.848343    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts" event={"ID":"d2ee09db-7efe-4b6a-8962-1c455bd9998c","Type":"ContainerStarted","Data":"7f582910ae060ce47d322e7623fa1262dbf6ce92b371767a579523487aee76c1"}
Sep 17 18:25:51.869831 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.869781    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts" podStartSLOduration=1.8697636800000002 podStartE2EDuration="1.86976368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:25:51.866882896 +0000 UTC m=+1387.546568751" watchObservedRunningTime="2026-09-17 18:25:51.86976368 +0000 UTC m=+1387.549449533"
Sep 17 18:25:51.986914 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:51.986878    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:52.827440 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:52.827401    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:52.854250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:52.854222    2589 generic.go:356] "Generic (PLEG): container finished" podID="d2ee09db-7efe-4b6a-8962-1c455bd9998c" containerID="fe54ef2e17ffaa25bd35681c6d32faad355a846aed990644baf2919fe74c4010" exitCode=0
Sep 17 18:25:52.854410 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:52.854303    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts" event={"ID":"d2ee09db-7efe-4b6a-8962-1c455bd9998c","Type":"ContainerDied","Data":"fe54ef2e17ffaa25bd35681c6d32faad355a846aed990644baf2919fe74c4010"}
Sep 17 18:25:53.732693 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:53.732647    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:53.983169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:53.983112    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"
Sep 17 18:25:54.145486 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:54.145455    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee09db-7efe-4b6a-8962-1c455bd9998c-maas-api-service-ca\") pod \"d2ee09db-7efe-4b6a-8962-1c455bd9998c\" (UID: \"d2ee09db-7efe-4b6a-8962-1c455bd9998c\") "
Sep 17 18:25:54.145822 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:54.145796    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2ee09db-7efe-4b6a-8962-1c455bd9998c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2ee09db-7efe-4b6a-8962-1c455bd9998c" (UID: "d2ee09db-7efe-4b6a-8962-1c455bd9998c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:25:54.246956 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:54.246877    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2ee09db-7efe-4b6a-8962-1c455bd9998c-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:25:54.525582 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:54.525550    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:54.865857 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:54.865775    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts" event={"ID":"d2ee09db-7efe-4b6a-8962-1c455bd9998c","Type":"ContainerDied","Data":"7f582910ae060ce47d322e7623fa1262dbf6ce92b371767a579523487aee76c1"}
Sep 17 18:25:54.865857 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:54.865805    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"
Sep 17 18:25:54.865857 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:54.865828    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f582910ae060ce47d322e7623fa1262dbf6ce92b371767a579523487aee76c1"
Sep 17 18:25:55.347347 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:55.347320    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:56.041694 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:56.041659    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:56.938315 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:56.938286    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:25:57.642117 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:25:57.642088    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:06.593440 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.593399    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:06.634718 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.634687    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"]
Sep 17 18:26:06.634950 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.634912    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2" podUID="f173e2c4-c0a1-4650-9870-d61191a4078b" containerName="payload-processing" containerID="cri-o://a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365" gracePeriod=30
Sep 17 18:26:06.645274 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.645245    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"]
Sep 17 18:26:06.645693 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.645505    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq" podUID="0b15b532-82ca-4f8f-a5b7-9695cc719737" containerName="payload-pre-processing" containerID="cri-o://c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853" gracePeriod=30
Sep 17 18:26:06.683924 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.683894    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:06.759648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.759611    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:06.867448 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.867353    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:06.873813 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.873777    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"]
Sep 17 18:26:06.886411 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.886382    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-d245d137-rt5ts"]
Sep 17 18:26:06.908265 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.908243    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:26:06.911369 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.911351    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:26:06.924783 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.924755    2589 generic.go:356] "Generic (PLEG): container finished" podID="0b15b532-82ca-4f8f-a5b7-9695cc719737" containerID="c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853" exitCode=0
Sep 17 18:26:06.924783 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.924780    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"
Sep 17 18:26:06.924939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.924834    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq" event={"ID":"0b15b532-82ca-4f8f-a5b7-9695cc719737","Type":"ContainerDied","Data":"c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853"}
Sep 17 18:26:06.924939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.924893    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq" event={"ID":"0b15b532-82ca-4f8f-a5b7-9695cc719737","Type":"ContainerDied","Data":"5e30aaddbd5ae2aaaae3e1dc60954c13e902e58dfba0d916915a5691e700feff"}
Sep 17 18:26:06.924939 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.924922    2589 scope.go:117] "RemoveContainer" containerID="c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853"
Sep 17 18:26:06.926321 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.926297    2589 generic.go:356] "Generic (PLEG): container finished" podID="f173e2c4-c0a1-4650-9870-d61191a4078b" containerID="a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365" exitCode=0
Sep 17 18:26:06.926486 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.926352    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"
Sep 17 18:26:06.926486 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.926381    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2" event={"ID":"f173e2c4-c0a1-4650-9870-d61191a4078b","Type":"ContainerDied","Data":"a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365"}
Sep 17 18:26:06.926486 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.926443    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2" event={"ID":"f173e2c4-c0a1-4650-9870-d61191a4078b","Type":"ContainerDied","Data":"3dd488a8aaa5c8b9a428b7b98cfe297bdecd99fb58367193cef7496c828c9fd8"}
Sep 17 18:26:06.934169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.934153    2589 scope.go:117] "RemoveContainer" containerID="c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853"
Sep 17 18:26:06.934425 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:26:06.934396    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853\": container with ID starting with c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853 not found: ID does not exist" containerID="c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853"
Sep 17 18:26:06.934499 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.934426    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853"} err="failed to get container status \"c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853\": rpc error: code = NotFound desc = could not find container \"c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853\": container with ID starting with c9a7de27065a0b8629ee27413e228f08858c1d80d311304f5a598ca344120853 not found: ID does not exist"
Sep 17 18:26:06.934499 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.934450    2589 scope.go:117] "RemoveContainer" containerID="a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365"
Sep 17 18:26:06.943861 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.943844    2589 scope.go:117] "RemoveContainer" containerID="a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365"
Sep 17 18:26:06.944177 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:26:06.944158    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365\": container with ID starting with a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365 not found: ID does not exist" containerID="a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365"
Sep 17 18:26:06.944254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.944187    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365"} err="failed to get container status \"a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365\": rpc error: code = NotFound desc = could not find container \"a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365\": container with ID starting with a80b48bbe8acfb050760572a479fe57a403e3eb241548625d93449aac2924365 not found: ID does not exist"
Sep 17 18:26:06.960977 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.960950    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mwmdd\" (UniqueName: \"kubernetes.io/projected/0b15b532-82ca-4f8f-a5b7-9695cc719737-kube-api-access-mwmdd\") pod \"0b15b532-82ca-4f8f-a5b7-9695cc719737\" (UID: \"0b15b532-82ca-4f8f-a5b7-9695cc719737\") "
Sep 17 18:26:06.961117 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.960992    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68mzk\" (UniqueName: \"kubernetes.io/projected/f173e2c4-c0a1-4650-9870-d61191a4078b-kube-api-access-68mzk\") pod \"f173e2c4-c0a1-4650-9870-d61191a4078b\" (UID: \"f173e2c4-c0a1-4650-9870-d61191a4078b\") "
Sep 17 18:26:06.961187 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.961107    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173e2c4-c0a1-4650-9870-d61191a4078b-plugins-config-volume\") pod \"f173e2c4-c0a1-4650-9870-d61191a4078b\" (UID: \"f173e2c4-c0a1-4650-9870-d61191a4078b\") "
Sep 17 18:26:06.961187 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.961160    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b532-82ca-4f8f-a5b7-9695cc719737-plugins-config-volume\") pod \"0b15b532-82ca-4f8f-a5b7-9695cc719737\" (UID: \"0b15b532-82ca-4f8f-a5b7-9695cc719737\") "
Sep 17 18:26:06.961571 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.961539    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f173e2c4-c0a1-4650-9870-d61191a4078b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f173e2c4-c0a1-4650-9870-d61191a4078b" (UID: "f173e2c4-c0a1-4650-9870-d61191a4078b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:06.961571 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.961560    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b15b532-82ca-4f8f-a5b7-9695cc719737-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b15b532-82ca-4f8f-a5b7-9695cc719737" (UID: "0b15b532-82ca-4f8f-a5b7-9695cc719737"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:26:06.963449 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.963409    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:06.963720 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.963673    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b15b532-82ca-4f8f-a5b7-9695cc719737-kube-api-access-mwmdd" (OuterVolumeSpecName: "kube-api-access-mwmdd") pod "0b15b532-82ca-4f8f-a5b7-9695cc719737" (UID: "0b15b532-82ca-4f8f-a5b7-9695cc719737"). InnerVolumeSpecName "kube-api-access-mwmdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:06.963908 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:06.963881    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f173e2c4-c0a1-4650-9870-d61191a4078b-kube-api-access-68mzk" (OuterVolumeSpecName: "kube-api-access-68mzk") pod "f173e2c4-c0a1-4650-9870-d61191a4078b" (UID: "f173e2c4-c0a1-4650-9870-d61191a4078b"). InnerVolumeSpecName "kube-api-access-68mzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:07.062493 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:07.062437    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f173e2c4-c0a1-4650-9870-d61191a4078b-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:07.062493 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:07.062490    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b15b532-82ca-4f8f-a5b7-9695cc719737-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:07.062493 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:07.062500    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwmdd\" (UniqueName: \"kubernetes.io/projected/0b15b532-82ca-4f8f-a5b7-9695cc719737-kube-api-access-mwmdd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:07.062493 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:07.062508    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68mzk\" (UniqueName: \"kubernetes.io/projected/f173e2c4-c0a1-4650-9870-d61191a4078b-kube-api-access-68mzk\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:07.252599 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:07.252566    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"]
Sep 17 18:26:07.257045 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:07.257021    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-d245d137-5c4c7b86d5-kjpb2"]
Sep 17 18:26:07.268225 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:07.268189    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"]
Sep 17 18:26:07.271698 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:07.271676    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-d245d137-856dcdb558-gm9tq"]
Sep 17 18:26:07.452519 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:07.452490    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:08.334924 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:08.334890    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:08.683452 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:08.683374    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b15b532-82ca-4f8f-a5b7-9695cc719737" path="/var/lib/kubelet/pods/0b15b532-82ca-4f8f-a5b7-9695cc719737/volumes"
Sep 17 18:26:08.683723 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:08.683710    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2ee09db-7efe-4b6a-8962-1c455bd9998c" path="/var/lib/kubelet/pods/d2ee09db-7efe-4b6a-8962-1c455bd9998c/volumes"
Sep 17 18:26:08.684043 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:08.684030    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f173e2c4-c0a1-4650-9870-d61191a4078b" path="/var/lib/kubelet/pods/f173e2c4-c0a1-4650-9870-d61191a4078b/volumes"
Sep 17 18:26:09.172385 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:09.172351    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:09.915139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:09.915100    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:10.015235 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:10.015209    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:10.721726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:10.721689    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:11.636197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:11.636162    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:12.398592 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:12.398543    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:14.965535 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:14.965501    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"]
Sep 17 18:26:15.062675 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:15.062642    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:18.966408 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:18.966370    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:19.066807 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:19.066778    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:21.548535 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548498    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"]
Sep 17 18:26:21.548907 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548836    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f173e2c4-c0a1-4650-9870-d61191a4078b" containerName="payload-processing"
Sep 17 18:26:21.548907 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548847    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="f173e2c4-c0a1-4650-9870-d61191a4078b" containerName="payload-processing"
Sep 17 18:26:21.548907 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548855    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b15b532-82ca-4f8f-a5b7-9695cc719737" containerName="payload-pre-processing"
Sep 17 18:26:21.548907 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548862    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b15b532-82ca-4f8f-a5b7-9695cc719737" containerName="payload-pre-processing"
Sep 17 18:26:21.548907 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548879    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2ee09db-7efe-4b6a-8962-1c455bd9998c" containerName="revoke-keys"
Sep 17 18:26:21.548907 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548885    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2ee09db-7efe-4b6a-8962-1c455bd9998c" containerName="revoke-keys"
Sep 17 18:26:21.549122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548942    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="f173e2c4-c0a1-4650-9870-d61191a4078b" containerName="payload-processing"
Sep 17 18:26:21.549122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548950    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2ee09db-7efe-4b6a-8962-1c455bd9998c" containerName="revoke-keys"
Sep 17 18:26:21.549122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.548960    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b15b532-82ca-4f8f-a5b7-9695cc719737" containerName="payload-pre-processing"
Sep 17 18:26:21.551788 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.551773    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.554753 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:26:21.554719    2589 status_manager.go:1018] "Failed to get status for pod" err="pods \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"e2e-models-e2e-worker-w4-b18433\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.554870 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:26:21.554789    2589 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"distinct-2-w4-b18433-kserve-self-signed-certs\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"e2e-models-e2e-worker-w4-b18433\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"distinct-2-w4-b18433-kserve-self-signed-certs\"" type="*v1.Secret"
Sep 17 18:26:21.568481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.568454    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"]
Sep 17 18:26:21.679755 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.679717    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:21.705854 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.705828    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tls-certs\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.705954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.705887    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-model-cache\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.705954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.705910    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-dshm\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.705954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.705950    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-home\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.706085 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.705966    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tmp-dir\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.706085 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.706030    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcspz\" (UniqueName: \"kubernetes.io/projected/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-kube-api-access-kcspz\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.807210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.807119    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-model-cache\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.807210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.807165    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-dshm\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.807210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.807191    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-home\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.807210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.807206    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tmp-dir\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.807499 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.807226    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcspz\" (UniqueName: \"kubernetes.io/projected/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-kube-api-access-kcspz\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.807499 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.807293    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tls-certs\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.807614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.807589    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-model-cache\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.807664 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.807600    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-home\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.807702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.807663    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tmp-dir\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.809660 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.809642    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-dshm\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.816238 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.816217    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcspz\" (UniqueName: \"kubernetes.io/projected/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-kube-api-access-kcspz\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:21.907828 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:21.907791    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:26:22.467402 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:22.467371    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-b18433\"/\"distinct-2-w4-b18433-kserve-self-signed-certs\""
Sep 17 18:26:22.469985 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:22.469965    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tls-certs\") pod \"distinct-2-w4-b18433-kserve-5b5d585f87-pszb7\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") " pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:22.760817 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:22.760781    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:23.056653 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:23.056481    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"]
Sep 17 18:26:23.059402 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:26:23.059368    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04dc3bf5_a77f_4790_ab6c_be5cd0a8b70a.slice/crio-6abc9df37adbc73045d50d471d5bb0a1200d7eb423952225958bce0b8d574b1e WatchSource:0}: Error finding container 6abc9df37adbc73045d50d471d5bb0a1200d7eb423952225958bce0b8d574b1e: Status 404 returned error can't find the container with id 6abc9df37adbc73045d50d471d5bb0a1200d7eb423952225958bce0b8d574b1e
Sep 17 18:26:24.018835 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:24.018797    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7" event={"ID":"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a","Type":"ContainerStarted","Data":"62c870347f922b55b947b60eb4e7789a25516f73462757f660e12cabf4d51937"}
Sep 17 18:26:24.019225 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:24.018843    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7" event={"ID":"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a","Type":"ContainerStarted","Data":"6abc9df37adbc73045d50d471d5bb0a1200d7eb423952225958bce0b8d574b1e"}
Sep 17 18:26:24.036748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:24.036700    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7" podStartSLOduration=3.036685829 podStartE2EDuration="3.036685829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:24.03610091 +0000 UTC m=+1419.715786768" watchObservedRunningTime="2026-09-17 18:26:24.036685829 +0000 UTC m=+1419.716371684"
Sep 17 18:26:29.987149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:29.987046    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps" podUID="b7a57a31-0ed7-419d-b59d-435bf124d15a" containerName="main" containerID="cri-o://c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698" gracePeriod=45
Sep 17 18:26:30.253211 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.253181    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:26:30.394489 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.394451    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-model-cache\") pod \"b7a57a31-0ed7-419d-b59d-435bf124d15a\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") "
Sep 17 18:26:30.394686 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.394496    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6pvhr\" (UniqueName: \"kubernetes.io/projected/b7a57a31-0ed7-419d-b59d-435bf124d15a-kube-api-access-6pvhr\") pod \"b7a57a31-0ed7-419d-b59d-435bf124d15a\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") "
Sep 17 18:26:30.394686 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.394529    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a57a31-0ed7-419d-b59d-435bf124d15a-tls-certs\") pod \"b7a57a31-0ed7-419d-b59d-435bf124d15a\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") "
Sep 17 18:26:30.394811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.394722    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-home\") pod \"b7a57a31-0ed7-419d-b59d-435bf124d15a\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") "
Sep 17 18:26:30.394811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.394731    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7a57a31-0ed7-419d-b59d-435bf124d15a" (UID: "b7a57a31-0ed7-419d-b59d-435bf124d15a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:30.394811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.394767    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-dshm\") pod \"b7a57a31-0ed7-419d-b59d-435bf124d15a\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") "
Sep 17 18:26:30.394946 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.394843    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-tmp-dir\") pod \"b7a57a31-0ed7-419d-b59d-435bf124d15a\" (UID: \"b7a57a31-0ed7-419d-b59d-435bf124d15a\") "
Sep 17 18:26:30.395031 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.394953    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-home" (OuterVolumeSpecName: "home") pod "b7a57a31-0ed7-419d-b59d-435bf124d15a" (UID: "b7a57a31-0ed7-419d-b59d-435bf124d15a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:30.395277 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.395183    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7a57a31-0ed7-419d-b59d-435bf124d15a" (UID: "b7a57a31-0ed7-419d-b59d-435bf124d15a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:30.395277 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.395198    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:30.395277 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.395215    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:30.396891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.396870    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7a57a31-0ed7-419d-b59d-435bf124d15a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7a57a31-0ed7-419d-b59d-435bf124d15a" (UID: "b7a57a31-0ed7-419d-b59d-435bf124d15a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:26:30.396976 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.396936    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-dshm" (OuterVolumeSpecName: "dshm") pod "b7a57a31-0ed7-419d-b59d-435bf124d15a" (UID: "b7a57a31-0ed7-419d-b59d-435bf124d15a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:26:30.397051 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.396994    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7a57a31-0ed7-419d-b59d-435bf124d15a-kube-api-access-6pvhr" (OuterVolumeSpecName: "kube-api-access-6pvhr") pod "b7a57a31-0ed7-419d-b59d-435bf124d15a" (UID: "b7a57a31-0ed7-419d-b59d-435bf124d15a"). InnerVolumeSpecName "kube-api-access-6pvhr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:26:30.496551 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.496468    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7a57a31-0ed7-419d-b59d-435bf124d15a-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:30.496551 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.496496    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:30.496551 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.496504    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7a57a31-0ed7-419d-b59d-435bf124d15a-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:30.496551 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:30.496513    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pvhr\" (UniqueName: \"kubernetes.io/projected/b7a57a31-0ed7-419d-b59d-435bf124d15a-kube-api-access-6pvhr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:26:31.056590 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:31.056552    2589 generic.go:356] "Generic (PLEG): container finished" podID="b7a57a31-0ed7-419d-b59d-435bf124d15a" containerID="c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698" exitCode=0
Sep 17 18:26:31.056590 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:31.056592    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"
Sep 17 18:26:31.057159 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:31.056640    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps" event={"ID":"b7a57a31-0ed7-419d-b59d-435bf124d15a","Type":"ContainerDied","Data":"c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698"}
Sep 17 18:26:31.057159 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:31.056678    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps" event={"ID":"b7a57a31-0ed7-419d-b59d-435bf124d15a","Type":"ContainerDied","Data":"01a39f12952b53979570698151d1cc6ff65f402d0fee1fe7fe3208323625bbe8"}
Sep 17 18:26:31.057159 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:31.056696    2589 scope.go:117] "RemoveContainer" containerID="c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698"
Sep 17 18:26:31.065465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:31.065448    2589 scope.go:117] "RemoveContainer" containerID="c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698"
Sep 17 18:26:31.065740 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:26:31.065723    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698\": container with ID starting with c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698 not found: ID does not exist" containerID="c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698"
Sep 17 18:26:31.065784 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:31.065749    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698"} err="failed to get container status \"c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698\": rpc error: code = NotFound desc = could not find container \"c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698\": container with ID starting with c572ab30dc6a38889221a330852975e622b1cd3753718e0464f1e4a68d0ea698 not found: ID does not exist"
Sep 17 18:26:31.072454 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:31.072432    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"]
Sep 17 18:26:31.076768 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:31.076741    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-d245d137/e2e-lifecycle-model-d245d137-kserve-5f5d899fd7-25dps"]
Sep 17 18:26:32.688437 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:32.688406    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7a57a31-0ed7-419d-b59d-435bf124d15a" path="/var/lib/kubelet/pods/b7a57a31-0ed7-419d-b59d-435bf124d15a/volumes"
Sep 17 18:26:32.761880 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:32.761849    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:32.761880 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:32.761884    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:32.774476 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:32.774446    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:33.078881 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:33.078853    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:26:44.591022 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.590978    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"]
Sep 17 18:26:44.591401 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.591316    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7a57a31-0ed7-419d-b59d-435bf124d15a" containerName="main"
Sep 17 18:26:44.591401 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.591327    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7a57a31-0ed7-419d-b59d-435bf124d15a" containerName="main"
Sep 17 18:26:44.591401 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.591397    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7a57a31-0ed7-419d-b59d-435bf124d15a" containerName="main"
Sep 17 18:26:44.593134 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.593114    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.596472 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.596451    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"kube-root-ca.crt\""
Sep 17 18:26:44.596472 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.596471    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"default-dockercfg-8sbhg\""
Sep 17 18:26:44.596630 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.596452    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"openshift-service-ca.crt\""
Sep 17 18:26:44.596630 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.596536    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"auth-test-model-d81d7c-kserve-self-signed-certs\""
Sep 17 18:26:44.603135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.603115    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"]
Sep 17 18:26:44.621952 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.621926    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:44.622984 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.622963    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-dshm\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.623112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.623036    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkq89\" (UniqueName: \"kubernetes.io/projected/66967875-8d1e-4036-9923-baf63df706d5-kube-api-access-xkq89\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.623112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.623071    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-model-cache\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.623201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.623160    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-home\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.623201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.623198    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-tmp-dir\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.623274 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.623220    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66967875-8d1e-4036-9923-baf63df706d5-tls-certs\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.724252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.724229    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkq89\" (UniqueName: \"kubernetes.io/projected/66967875-8d1e-4036-9923-baf63df706d5-kube-api-access-xkq89\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.724371 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.724259    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-model-cache\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.724371 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.724298    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-home\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.724371 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.724330    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-tmp-dir\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.724371 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.724357    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66967875-8d1e-4036-9923-baf63df706d5-tls-certs\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.724576 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.724458    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-dshm\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.724708 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.724687    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-model-cache\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.724774 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.724743    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-tmp-dir\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.724774 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.724759    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-home\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.726885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.726860    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-dshm\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.727139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.727124    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66967875-8d1e-4036-9923-baf63df706d5-tls-certs\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.732633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.732617    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkq89\" (UniqueName: \"kubernetes.io/projected/66967875-8d1e-4036-9923-baf63df706d5-kube-api-access-xkq89\") pod \"auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:44.903393 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:44.903306    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:45.176421 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:45.176395    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"]
Sep 17 18:26:45.178736 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:26:45.178701    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66967875_8d1e_4036_9923_baf63df706d5.slice/crio-ee976d95d4b1abdbe4f6c35142897fb0674490ec8fe2460267c80c0e51ae7322 WatchSource:0}: Error finding container ee976d95d4b1abdbe4f6c35142897fb0674490ec8fe2460267c80c0e51ae7322: Status 404 returned error can't find the container with id ee976d95d4b1abdbe4f6c35142897fb0674490ec8fe2460267c80c0e51ae7322
Sep 17 18:26:46.132720 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:46.132685    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj" event={"ID":"66967875-8d1e-4036-9923-baf63df706d5","Type":"ContainerStarted","Data":"060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961"}
Sep 17 18:26:46.132720 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:46.132727    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj" event={"ID":"66967875-8d1e-4036-9923-baf63df706d5","Type":"ContainerStarted","Data":"ee976d95d4b1abdbe4f6c35142897fb0674490ec8fe2460267c80c0e51ae7322"}
Sep 17 18:26:46.150975 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:46.150932    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj" podStartSLOduration=2.150916917 podStartE2EDuration="2.150916917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:26:46.149980135 +0000 UTC m=+1441.829665988" watchObservedRunningTime="2026-09-17 18:26:46.150916917 +0000 UTC m=+1441.830602802"
Sep 17 18:26:46.399419 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:46.399326    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:26:49.650782 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:49.650751    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:51.440802 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:51.440760    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:52.050553 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:52.050521    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:52.830926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:52.830893    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:26:54.903891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:54.903859    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:54.904278 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:54.903906    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:54.916269 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:54.916244    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:55.188124 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:55.188033    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:26:58.899955 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:26:58.899913    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:05.169048 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:05.168994    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:21.017752 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:21.017719    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:27:27.261317 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:27.261280    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:32.743312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:32.743266    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:32.846024 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:32.845979    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:32.996021 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:32.995926    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"]
Sep 17 18:27:32.996183 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:32.996161    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy" containerID="cri-o://ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88" gracePeriod=30
Sep 17 18:27:33.023368 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:33.023331    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:34.202851 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:34.202815    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:34.303182 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:34.303151    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:34.401718 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:34.401684    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:35.014712 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:35.014681    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:35.198231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:35.198194    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:36.399977 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:36.399948    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:38.247331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.247299    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:27:38.331950 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.331876    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-data\") pod \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") "
Sep 17 18:27:38.332133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.331967    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-certs\") pod \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") "
Sep 17 18:27:38.332133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332035    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-envoy\") pod \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") "
Sep 17 18:27:38.332133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332071    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-podinfo\") pod \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") "
Sep 17 18:27:38.332133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332129    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-socket\") pod \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") "
Sep 17 18:27:38.332311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332194    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-data" (OuterVolumeSpecName: "istio-data") pod "22b1304e-a4b4-4d8e-b186-b6c0623e6953" (UID: "22b1304e-a4b4-4d8e-b186-b6c0623e6953"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:38.332311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332244    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22b1304e-a4b4-4d8e-b186-b6c0623e6953" (UID: "22b1304e-a4b4-4d8e-b186-b6c0623e6953"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:38.332311 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332254    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-token\") pod \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") "
Sep 17 18:27:38.332444 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332318    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4dfs9\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-kube-api-access-4dfs9\") pod \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") "
Sep 17 18:27:38.332444 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332369    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-credential-socket\") pod \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") "
Sep 17 18:27:38.332444 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332397    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istiod-ca-cert\") pod \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\" (UID: \"22b1304e-a4b4-4d8e-b186-b6c0623e6953\") "
Sep 17 18:27:38.332444 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332407    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22b1304e-a4b4-4d8e-b186-b6c0623e6953" (UID: "22b1304e-a4b4-4d8e-b186-b6c0623e6953"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:38.332660 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332598    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22b1304e-a4b4-4d8e-b186-b6c0623e6953" (UID: "22b1304e-a4b4-4d8e-b186-b6c0623e6953"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:38.332949 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332854    2589 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:27:38.333045 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332951    2589 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:27:38.333045 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332964    2589 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:27:38.333045 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.332978    2589 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:27:38.333200 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.333065    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22b1304e-a4b4-4d8e-b186-b6c0623e6953" (UID: "22b1304e-a4b4-4d8e-b186-b6c0623e6953"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:27:38.334409 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.334377    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22b1304e-a4b4-4d8e-b186-b6c0623e6953" (UID: "22b1304e-a4b4-4d8e-b186-b6c0623e6953"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:27:38.334856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.334835    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-kube-api-access-4dfs9" (OuterVolumeSpecName: "kube-api-access-4dfs9") pod "22b1304e-a4b4-4d8e-b186-b6c0623e6953" (UID: "22b1304e-a4b4-4d8e-b186-b6c0623e6953"). InnerVolumeSpecName "kube-api-access-4dfs9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:38.334964 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.334944    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22b1304e-a4b4-4d8e-b186-b6c0623e6953" (UID: "22b1304e-a4b4-4d8e-b186-b6c0623e6953"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:27:38.335215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.335195    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-token" (OuterVolumeSpecName: "istio-token") pod "22b1304e-a4b4-4d8e-b186-b6c0623e6953" (UID: "22b1304e-a4b4-4d8e-b186-b6c0623e6953"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:27:38.398149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.398110    2589 generic.go:356] "Generic (PLEG): container finished" podID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerID="ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88" exitCode=0
Sep 17 18:27:38.398297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.398194    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"
Sep 17 18:27:38.398297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.398195    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" event={"ID":"22b1304e-a4b4-4d8e-b186-b6c0623e6953","Type":"ContainerDied","Data":"ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88"}
Sep 17 18:27:38.398414 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.398349    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw" event={"ID":"22b1304e-a4b4-4d8e-b186-b6c0623e6953","Type":"ContainerDied","Data":"50ace6c56747cc51de438b6b1d60ff278a72ee357da29a9badf65df2a91906f6"}
Sep 17 18:27:38.398414 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.398373    2589 scope.go:117] "RemoveContainer" containerID="ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88"
Sep 17 18:27:38.410151 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.410128    2589 scope.go:117] "RemoveContainer" containerID="faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455"
Sep 17 18:27:38.418683 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.418663    2589 scope.go:117] "RemoveContainer" containerID="ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88"
Sep 17 18:27:38.419111 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:27:38.419070    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88\": container with ID starting with ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88 not found: ID does not exist" containerID="ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88"
Sep 17 18:27:38.419222 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.419136    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88"} err="failed to get container status \"ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88\": rpc error: code = NotFound desc = could not find container \"ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88\": container with ID starting with ada9267204e59fca71a155a0635674430251602134355a0f6b243ec5f4398a88 not found: ID does not exist"
Sep 17 18:27:38.419222 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.419163    2589 scope.go:117] "RemoveContainer" containerID="faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455"
Sep 17 18:27:38.419442 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:27:38.419426    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455\": container with ID starting with faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455 not found: ID does not exist" containerID="faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455"
Sep 17 18:27:38.419488 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.419448    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455"} err="failed to get container status \"faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455\": rpc error: code = NotFound desc = could not find container \"faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455\": container with ID starting with faf4253fc616e900dde4546449e5839258c3202a37eb1ad25210fbf20de58455 not found: ID does not exist"
Sep 17 18:27:38.420989 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.420971    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"]
Sep 17 18:27:38.424503 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.424483    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-d245d137-openshift-default-c948444cb-ggjfw"]
Sep 17 18:27:38.434118 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.434096    2589 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:27:38.434118 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.434118    2589 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:27:38.434225 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.434127    2589 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:27:38.434225 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.434136    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dfs9\" (UniqueName: \"kubernetes.io/projected/22b1304e-a4b4-4d8e-b186-b6c0623e6953-kube-api-access-4dfs9\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:27:38.434225 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.434145    2589 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b1304e-a4b4-4d8e-b186-b6c0623e6953-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:27:38.684120 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:38.684039    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" path="/var/lib/kubelet/pods/22b1304e-a4b4-4d8e-b186-b6c0623e6953/volumes"
Sep 17 18:27:42.800755 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:42.800719    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:42.902064 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:42.902036    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:43.466941 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:43.466909    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:46.664719 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:46.664691    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:27:46.665143 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:46.664974    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:27:46.792422 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:46.792394    2589 scope.go:117] "RemoveContainer" containerID="cbd9e05d08d62496d8c84240c367a2cd5b70a52d2e4fb40145391e2a2f0b6f87"
Sep 17 18:27:46.800827 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:46.800807    2589 scope.go:117] "RemoveContainer" containerID="72ea62885a65017c27a90d709d9d032a36c10cf264170c44c896a3ea5fb6757b"
Sep 17 18:27:46.808732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:46.808715    2589 scope.go:117] "RemoveContainer" containerID="bbce35ee340da13969245d13280393551e5a24a1e2a9c5c180ecfd0503483656"
Sep 17 18:27:48.083101 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:48.083066    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:27:50.097123 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:50.097087    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:50.196227 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:50.196191    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:27:50.294516 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:27:50.294483    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:00.770563 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:00.770470    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:00.991598 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:00.991561    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:01.550185 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:01.550143    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:08.330793 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:08.330762    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:22.616213 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:22.616173    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:25.594524 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:25.594492    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:28:28.353394 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:28.353358    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:28.452712 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:28.452685    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:28.712576 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:28.712484    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:29.027840 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.027808    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"]
Sep 17 18:28:29.028187 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.028174    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy"
Sep 17 18:28:29.028249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.028189    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy"
Sep 17 18:28:29.028249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.028200    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy"
Sep 17 18:28:29.028249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.028205    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy"
Sep 17 18:28:29.028359 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.028261    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy"
Sep 17 18:28:29.028359 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.028268    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="22b1304e-a4b4-4d8e-b186-b6c0623e6953" containerName="istio-proxy"
Sep 17 18:28:29.031132 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.031116    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.033742 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.033707    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-4faba0-openshift-default-dockercfg-q2x66\""
Sep 17 18:28:29.044988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.044962    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:29.046218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.046198    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"]
Sep 17 18:28:29.116988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.116959    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-credential-socket\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.117168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.116996    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-envoy\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.117168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.117079    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-socket\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.117168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.117113    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-data\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.117168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.117135    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-certs\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.117168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.117166    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s8s7\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-kube-api-access-4s8s7\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.117339 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.117204    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33871cf8-346a-47be-9aee-21d073db84d0-istiod-ca-cert\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.117339 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.117223    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33871cf8-346a-47be-9aee-21d073db84d0-istio-podinfo\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.117339 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.117242    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-istio-token\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.142132 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.142111    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:29.218500 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.218466    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-certs\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.218671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.218515    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s8s7\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-kube-api-access-4s8s7\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.218671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.218544    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33871cf8-346a-47be-9aee-21d073db84d0-istiod-ca-cert\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.218671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.218564    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33871cf8-346a-47be-9aee-21d073db84d0-istio-podinfo\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.218671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.218583    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-istio-token\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.218671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.218653    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-credential-socket\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.218930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.218691    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-envoy\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.218930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.218732    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-socket\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.218930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.218757    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-data\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.219112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.219048    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-certs\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.219112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.219062    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-credential-socket\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.219222 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.219163    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-data\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.219222 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.219179    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-socket\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.219457 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.219438    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33871cf8-346a-47be-9aee-21d073db84d0-istiod-ca-cert\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.221173 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.221155    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-envoy\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.221390 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.221374    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33871cf8-346a-47be-9aee-21d073db84d0-istio-podinfo\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.226911 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.226888    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-istio-token\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.227033 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.226940    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s8s7\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-kube-api-access-4s8s7\") pod \"e2e-api-a-4faba0-openshift-default-55b947b884-8h457\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") " pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.341259 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.341159    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:29.616488 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.616462    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"]
Sep 17 18:28:29.619086 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:28:29.619055    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33871cf8_346a_47be_9aee_21d073db84d0.slice/crio-d864afe6bece9acd9f81c09198da31efa4b33ddcd8fab0b94834d5920a04616f WatchSource:0}: Error finding container d864afe6bece9acd9f81c09198da31efa4b33ddcd8fab0b94834d5920a04616f: Status 404 returned error can't find the container with id d864afe6bece9acd9f81c09198da31efa4b33ddcd8fab0b94834d5920a04616f
Sep 17 18:28:29.621474 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.621440    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:28:29.621580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.621522    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:28:29.621580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.621572    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:28:29.653240 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:29.653212    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" event={"ID":"33871cf8-346a-47be-9aee-21d073db84d0","Type":"ContainerStarted","Data":"d864afe6bece9acd9f81c09198da31efa4b33ddcd8fab0b94834d5920a04616f"}
Sep 17 18:28:30.659681 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:30.659648    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" event={"ID":"33871cf8-346a-47be-9aee-21d073db84d0","Type":"ContainerStarted","Data":"6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30"}
Sep 17 18:28:30.682182 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:30.682140    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" podStartSLOduration=1.682124309 podStartE2EDuration="1.682124309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:30.67985561 +0000 UTC m=+1546.359541466" watchObservedRunningTime="2026-09-17 18:28:30.682124309 +0000 UTC m=+1546.361810163"
Sep 17 18:28:31.341959 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:31.341921    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:31.347185 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:31.347159    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:31.664716 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:31.664633    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:31.665835 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:31.665812    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:28:32.530432 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.530402    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:32.632355 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.632328    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:32.731703 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.731662    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:32.891611 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.891533    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"]
Sep 17 18:28:32.894740 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.894724    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:32.897140 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.897116    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-5d8053\"/\"embedding-w0-5d8053-kserve-self-signed-certs\""
Sep 17 18:28:32.907110 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.907091    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"]
Sep 17 18:28:32.954767 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.954734    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-home\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:32.954934 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.954816    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-model-cache\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:32.954934 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.954843    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533e9753-039d-44b4-bada-b6042f56cac9-tls-certs\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:32.954934 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.954906    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-tmp-dir\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:32.955116 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.954940    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-dshm\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:32.955116 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:32.955029    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pfxm\" (UniqueName: \"kubernetes.io/projected/533e9753-039d-44b4-bada-b6042f56cac9-kube-api-access-4pfxm\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.017852 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.017819    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:33.056038 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.055987    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-model-cache\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.056190 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.056045    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533e9753-039d-44b4-bada-b6042f56cac9-tls-certs\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.056190 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.056092    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-tmp-dir\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.056190 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.056127    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-dshm\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.056350 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.056256    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pfxm\" (UniqueName: \"kubernetes.io/projected/533e9753-039d-44b4-bada-b6042f56cac9-kube-api-access-4pfxm\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.056411 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.056364    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-home\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.056465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.056411    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-model-cache\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.056518 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.056462    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-tmp-dir\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.056714 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.056689    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-home\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.058604 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.058583    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-dshm\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.058811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.058797    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533e9753-039d-44b4-bada-b6042f56cac9-tls-certs\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.082919 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.082889    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pfxm\" (UniqueName: \"kubernetes.io/projected/533e9753-039d-44b4-bada-b6042f56cac9-kube-api-access-4pfxm\") pod \"embedding-w0-5d8053-kserve-6786f687f9-gqt8g\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") " pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.204549 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.204459    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:33.572385 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.572357    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"]
Sep 17 18:28:33.574421 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:28:33.574394    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod533e9753_039d_44b4_bada_b6042f56cac9.slice/crio-f9a66931205015224f861b67ca384faa6c2c8d7e9723993a2584202d4ecfbf54 WatchSource:0}: Error finding container f9a66931205015224f861b67ca384faa6c2c8d7e9723993a2584202d4ecfbf54: Status 404 returned error can't find the container with id f9a66931205015224f861b67ca384faa6c2c8d7e9723993a2584202d4ecfbf54
Sep 17 18:28:33.674900 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.674869    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g" event={"ID":"533e9753-039d-44b4-bada-b6042f56cac9","Type":"ContainerStarted","Data":"1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934"}
Sep 17 18:28:33.675089 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.674915    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g" event={"ID":"533e9753-039d-44b4-bada-b6042f56cac9","Type":"ContainerStarted","Data":"f9a66931205015224f861b67ca384faa6c2c8d7e9723993a2584202d4ecfbf54"}
Sep 17 18:28:33.729671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.729617    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g" podStartSLOduration=1.729598805 podStartE2EDuration="1.729598805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:33.726844293 +0000 UTC m=+1549.406530148" watchObservedRunningTime="2026-09-17 18:28:33.729598805 +0000 UTC m=+1549.409284658"
Sep 17 18:28:33.958078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:33.957980    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:34.873440 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:34.873409    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:35.734376 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:35.734343    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:36.619443 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:36.619410    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:37.551404 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:37.551372    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:38.376975 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:38.376944    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:38.477807 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:38.477771    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:38.582683 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:38.582651    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:38.880250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:38.880218    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:38.998397 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:38.998370    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:28:39.081598 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.081566    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"]
Sep 17 18:28:39.084996 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.084980    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:39.087293 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.087271    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4faba0\""
Sep 17 18:28:39.087428 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.087369    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4faba0-dockercfg-8b7mg\""
Sep 17 18:28:39.098656 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.098633    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"]
Sep 17 18:28:39.177640 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.177569    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"]
Sep 17 18:28:39.180926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.180907    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:28:39.193154 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.193131    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"]
Sep 17 18:28:39.216086 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.216055    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7glx\" (UniqueName: \"kubernetes.io/projected/00a3a067-e994-4932-83ca-88a14d3e9054-kube-api-access-n7glx\") pod \"payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb\" (UID: \"00a3a067-e994-4932-83ca-88a14d3e9054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:39.216216 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.216198    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a3a067-e994-4932-83ca-88a14d3e9054-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb\" (UID: \"00a3a067-e994-4932-83ca-88a14d3e9054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:39.317331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.317303    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7glx\" (UniqueName: \"kubernetes.io/projected/00a3a067-e994-4932-83ca-88a14d3e9054-kube-api-access-n7glx\") pod \"payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb\" (UID: \"00a3a067-e994-4932-83ca-88a14d3e9054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:39.317446 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.317381    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv\" (UID: \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:28:39.317446 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.317427    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn5tv\" (UniqueName: \"kubernetes.io/projected/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-kube-api-access-hn5tv\") pod \"payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv\" (UID: \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:28:39.317544 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.317460    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a3a067-e994-4932-83ca-88a14d3e9054-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb\" (UID: \"00a3a067-e994-4932-83ca-88a14d3e9054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:39.318149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.318133    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a3a067-e994-4932-83ca-88a14d3e9054-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb\" (UID: \"00a3a067-e994-4932-83ca-88a14d3e9054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:39.325326 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.325302    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7glx\" (UniqueName: \"kubernetes.io/projected/00a3a067-e994-4932-83ca-88a14d3e9054-kube-api-access-n7glx\") pod \"payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb\" (UID: \"00a3a067-e994-4932-83ca-88a14d3e9054\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:39.396028 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.395985    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:39.418324 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.418290    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv\" (UID: \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:28:39.418452 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.418362    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn5tv\" (UniqueName: \"kubernetes.io/projected/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-kube-api-access-hn5tv\") pod \"payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv\" (UID: \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:28:39.419265 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.419240    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv\" (UID: \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:28:39.426301 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.426276    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn5tv\" (UniqueName: \"kubernetes.io/projected/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-kube-api-access-hn5tv\") pod \"payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv\" (UID: \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\") " pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:28:39.490814 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.490777    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:28:39.671319 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.671285    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"]
Sep 17 18:28:39.672253 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:28:39.672225    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a3a067_e994_4932_83ca_88a14d3e9054.slice/crio-467f959cf98895f0aab761fd995acfe80b5e19846577949250b01e7603592dbc WatchSource:0}: Error finding container 467f959cf98895f0aab761fd995acfe80b5e19846577949250b01e7603592dbc: Status 404 returned error can't find the container with id 467f959cf98895f0aab761fd995acfe80b5e19846577949250b01e7603592dbc
Sep 17 18:28:39.704564 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.704507    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb" event={"ID":"00a3a067-e994-4932-83ca-88a14d3e9054","Type":"ContainerStarted","Data":"467f959cf98895f0aab761fd995acfe80b5e19846577949250b01e7603592dbc"}
Sep 17 18:28:39.768200 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:39.768177    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"]
Sep 17 18:28:39.769735 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:28:39.769710    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf34e9c_af10_4b36_af49_aa0b9d3a9ff0.slice/crio-1ddf8c4236534d687e1fc2be93fe2c24fa961c7a67cd39a19d0d2e64ca42bdc3 WatchSource:0}: Error finding container 1ddf8c4236534d687e1fc2be93fe2c24fa961c7a67cd39a19d0d2e64ca42bdc3: Status 404 returned error can't find the container with id 1ddf8c4236534d687e1fc2be93fe2c24fa961c7a67cd39a19d0d2e64ca42bdc3
Sep 17 18:28:40.712075 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:40.712040    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb" event={"ID":"00a3a067-e994-4932-83ca-88a14d3e9054","Type":"ContainerStarted","Data":"09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a"}
Sep 17 18:28:40.712540 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:40.712101    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:40.713964 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:40.713930    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv" event={"ID":"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0","Type":"ContainerStarted","Data":"3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be"}
Sep 17 18:28:40.713964 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:40.713967    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv" event={"ID":"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0","Type":"ContainerStarted","Data":"1ddf8c4236534d687e1fc2be93fe2c24fa961c7a67cd39a19d0d2e64ca42bdc3"}
Sep 17 18:28:40.714189 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:40.713988    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:28:40.732112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:40.732068    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb" podStartSLOduration=1.732053126 podStartE2EDuration="1.732053126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:40.730115383 +0000 UTC m=+1556.409801238" watchObservedRunningTime="2026-09-17 18:28:40.732053126 +0000 UTC m=+1556.411738979"
Sep 17 18:28:40.752289 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:40.752242    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv" podStartSLOduration=1.752226178 podStartE2EDuration="1.752226178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:28:40.751064373 +0000 UTC m=+1556.430750227" watchObservedRunningTime="2026-09-17 18:28:40.752226178 +0000 UTC m=+1556.431912036"
Sep 17 18:28:43.204972 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:43.204938    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:43.205419 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:43.204987    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:43.217662 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:43.217635    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:43.739150 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:43.739117    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:28:51.720712 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:51.720680    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:28:51.721200 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:28:51.720753    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:29:00.517604 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:00.517554    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:01.666068 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:01.665964    2589 patch_prober.go:28] interesting pod/e2e-api-a-4faba0-openshift-default-55b947b884-8h457 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 17 18:29:01.666068 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:01.666039    2589 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 18:29:01.833418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:01.833385    2589 generic.go:356] "Generic (PLEG): container finished" podID="33871cf8-346a-47be-9aee-21d073db84d0" containerID="6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30" exitCode=0
Sep 17 18:29:01.833593 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:01.833467    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" event={"ID":"33871cf8-346a-47be-9aee-21d073db84d0","Type":"ContainerDied","Data":"6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30"}
Sep 17 18:29:01.833891 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:01.833876    2589 scope.go:117] "RemoveContainer" containerID="6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30"
Sep 17 18:29:01.835265 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:01.835238    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:29:01.835333 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:01.835305    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:29:01.835370 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:01.835339    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:29:02.004813 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:02.004777    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:02.106192 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:02.106152    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:02.704406 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:02.704364    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:02.797239 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:02.797203    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:02.840260 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:02.840225    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" event={"ID":"33871cf8-346a-47be-9aee-21d073db84d0","Type":"ContainerStarted","Data":"8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43"}
Sep 17 18:29:03.158017 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:03.157969    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:03.342197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:03.342161    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:29:03.342453 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:03.342428    2589 patch_prober.go:28] interesting pod/e2e-api-a-4faba0-openshift-default-55b947b884-8h457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 17 18:29:03.342508 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:03.342484    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 18:29:03.476424 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:03.476326    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:03.647937 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:03.647906    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:03.754096 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:03.754066    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:04.342388 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:04.342352    2589 patch_prober.go:28] interesting pod/e2e-api-a-4faba0-openshift-default-55b947b884-8h457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 17 18:29:04.342609 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:04.342414    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 18:29:04.575425 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:04.575389    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:05.342348 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:05.342311    2589 patch_prober.go:28] interesting pod/e2e-api-a-4faba0-openshift-default-55b947b884-8h457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 17 18:29:05.342812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:05.342370    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 18:29:05.420680 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:05.420644    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:06.227312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:06.227272    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:06.341603 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:06.341571    2589 patch_prober.go:28] interesting pod/e2e-api-a-4faba0-openshift-default-55b947b884-8h457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 17 18:29:06.341806 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:06.341633    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 18:29:07.032643 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:07.032603    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:07.124214 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:07.124179    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:07.342065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:07.341968    2589 patch_prober.go:28] interesting pod/e2e-api-a-4faba0-openshift-default-55b947b884-8h457 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 17 18:29:07.342217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:07.342060    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 18:29:07.936040 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:07.935979    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:08.346022 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:08.345983    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:29:08.346502 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:08.346429    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:29:08.347063 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:08.347044    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:29:08.825543 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:08.825505    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:09.602217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:09.602180    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:10.321233 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:10.321205    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:11.127349 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:11.127316    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:11.828651 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:11.828617    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:12.019046 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:12.018980    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:12.542090 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:12.542056    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:12.640533 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:12.640496    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:12.742156 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:12.742120    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:13.023888 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.023855    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:13.139417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.139383    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:13.186237 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.186201    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"]
Sep 17 18:29:13.189072 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.189051    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:13.191887 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.191855    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b06908\""
Sep 17 18:29:13.214894 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.214866    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"]
Sep 17 18:29:13.264778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.264744    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6srvp\" (UniqueName: \"kubernetes.io/projected/1509c79d-9c57-4cb2-80cc-94a55a0d3103-kube-api-access-6srvp\") pod \"payload-processing-e2e-api-b-b06908-64c748b995-9qfh6\" (UID: \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\") " pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:13.264778 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.264780    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1509c79d-9c57-4cb2-80cc-94a55a0d3103-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b06908-64c748b995-9qfh6\" (UID: \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\") " pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:13.365679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.365594    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6srvp\" (UniqueName: \"kubernetes.io/projected/1509c79d-9c57-4cb2-80cc-94a55a0d3103-kube-api-access-6srvp\") pod \"payload-processing-e2e-api-b-b06908-64c748b995-9qfh6\" (UID: \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\") " pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:13.365679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.365636    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1509c79d-9c57-4cb2-80cc-94a55a0d3103-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b06908-64c748b995-9qfh6\" (UID: \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\") " pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:13.366324 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.366304    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1509c79d-9c57-4cb2-80cc-94a55a0d3103-plugins-config-volume\") pod \"payload-processing-e2e-api-b-b06908-64c748b995-9qfh6\" (UID: \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\") " pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:13.373843 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.373819    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6srvp\" (UniqueName: \"kubernetes.io/projected/1509c79d-9c57-4cb2-80cc-94a55a0d3103-kube-api-access-6srvp\") pod \"payload-processing-e2e-api-b-b06908-64c748b995-9qfh6\" (UID: \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\") " pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:13.501646 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.501614    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:13.780465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.780439    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"]
Sep 17 18:29:13.782610 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:29:13.782579    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1509c79d_9c57_4cb2_80cc_94a55a0d3103.slice/crio-0ecb78e7c9cdba06192c42a4c8e58658439640afc8035a078205d6157338db83 WatchSource:0}: Error finding container 0ecb78e7c9cdba06192c42a4c8e58658439640afc8035a078205d6157338db83: Status 404 returned error can't find the container with id 0ecb78e7c9cdba06192c42a4c8e58658439640afc8035a078205d6157338db83
Sep 17 18:29:13.897059 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.896987    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6" event={"ID":"1509c79d-9c57-4cb2-80cc-94a55a0d3103","Type":"ContainerStarted","Data":"94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710"}
Sep 17 18:29:13.897258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.897079    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6" event={"ID":"1509c79d-9c57-4cb2-80cc-94a55a0d3103","Type":"ContainerStarted","Data":"0ecb78e7c9cdba06192c42a4c8e58658439640afc8035a078205d6157338db83"}
Sep 17 18:29:13.897258 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.897123    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:13.916981 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:13.916928    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6" podStartSLOduration=0.916913849 podStartE2EDuration="916.913849ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:13.915057114 +0000 UTC m=+1589.594742972" watchObservedRunningTime="2026-09-17 18:29:13.916913849 +0000 UTC m=+1589.596599702"
Sep 17 18:29:15.611222 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:15.611190    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:29:18.689910 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:18.689877    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:18.791588 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:18.791552    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:24.903510 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:24.903480    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:29:34.927122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:34.927032    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:35.825710 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:35.825673    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:36.635312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:36.635280    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:36.913499 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:36.913411    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:37.370650 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:37.370613    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:37.965752 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:37.965722    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:38.197031 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:38.196972    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:38.973812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:38.973772    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:39.783580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:39.783542    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"]
Sep 17 18:29:39.787234 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:39.787217    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"
Sep 17 18:29:39.789396 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:39.789376    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:29:39.789512 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:39.789383    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:29:39.795478 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:39.795454    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"]
Sep 17 18:29:39.824340 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:39.824304    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058bac6f-7fb2-4da0-8c74-9bf6160782b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r\" (UID: \"058bac6f-7fb2-4da0-8c74-9bf6160782b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"
Sep 17 18:29:39.841705 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:39.841666    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:39.925232 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:39.925199    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058bac6f-7fb2-4da0-8c74-9bf6160782b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r\" (UID: \"058bac6f-7fb2-4da0-8c74-9bf6160782b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"
Sep 17 18:29:39.925838 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:39.925818    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058bac6f-7fb2-4da0-8c74-9bf6160782b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r\" (UID: \"058bac6f-7fb2-4da0-8c74-9bf6160782b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"
Sep 17 18:29:40.098482 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:40.098394    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"
Sep 17 18:29:40.368932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:40.368906    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"]
Sep 17 18:29:40.371232 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:29:40.371207    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod058bac6f_7fb2_4da0_8c74_9bf6160782b4.slice/crio-7a984540141d74a568f853903598d0c5e60883f73de5f987d001009dc846a00d WatchSource:0}: Error finding container 7a984540141d74a568f853903598d0c5e60883f73de5f987d001009dc846a00d: Status 404 returned error can't find the container with id 7a984540141d74a568f853903598d0c5e60883f73de5f987d001009dc846a00d
Sep 17 18:29:40.606349 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:40.606318    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:41.034821 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:41.034788    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r" event={"ID":"058bac6f-7fb2-4da0-8c74-9bf6160782b4","Type":"ContainerStarted","Data":"713c2f80c725d5c2ba9305aed68485b6322c5a4beb22340aca1d350cf5e34640"}
Sep 17 18:29:41.034992 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:41.034864    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r" event={"ID":"058bac6f-7fb2-4da0-8c74-9bf6160782b4","Type":"ContainerStarted","Data":"7a984540141d74a568f853903598d0c5e60883f73de5f987d001009dc846a00d"}
Sep 17 18:29:41.053048 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:41.052979    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r" podStartSLOduration=2.052964716 podStartE2EDuration="2.052964716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:29:41.051747438 +0000 UTC m=+1616.731433292" watchObservedRunningTime="2026-09-17 18:29:41.052964716 +0000 UTC m=+1616.732650600"
Sep 17 18:29:41.296488 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:41.296379    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:42.041648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:42.041602    2589 generic.go:356] "Generic (PLEG): container finished" podID="058bac6f-7fb2-4da0-8c74-9bf6160782b4" containerID="713c2f80c725d5c2ba9305aed68485b6322c5a4beb22340aca1d350cf5e34640" exitCode=0
Sep 17 18:29:42.041856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:42.041648    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r" event={"ID":"058bac6f-7fb2-4da0-8c74-9bf6160782b4","Type":"ContainerDied","Data":"713c2f80c725d5c2ba9305aed68485b6322c5a4beb22340aca1d350cf5e34640"}
Sep 17 18:29:42.041856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:42.041709    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:42.764346 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:42.764314    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:42.949905 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:42.949868    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"]
Sep 17 18:29:42.981909 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:42.981870    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"]
Sep 17 18:29:43.044379 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:43.044286    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:43.181207 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:43.181184    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"
Sep 17 18:29:43.256263 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:43.256234    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058bac6f-7fb2-4da0-8c74-9bf6160782b4-maas-api-service-ca\") pod \"058bac6f-7fb2-4da0-8c74-9bf6160782b4\" (UID: \"058bac6f-7fb2-4da0-8c74-9bf6160782b4\") "
Sep 17 18:29:43.256619 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:43.256595    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/058bac6f-7fb2-4da0-8c74-9bf6160782b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "058bac6f-7fb2-4da0-8c74-9bf6160782b4" (UID: "058bac6f-7fb2-4da0-8c74-9bf6160782b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:43.357580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:43.357504    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/058bac6f-7fb2-4da0-8c74-9bf6160782b4-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:43.636756 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:43.636672    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:44.053553 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:44.053525    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"
Sep 17 18:29:44.053981 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:44.053523    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r" event={"ID":"058bac6f-7fb2-4da0-8c74-9bf6160782b4","Type":"ContainerDied","Data":"7a984540141d74a568f853903598d0c5e60883f73de5f987d001009dc846a00d"}
Sep 17 18:29:44.053981 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:44.053643    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a984540141d74a568f853903598d0c5e60883f73de5f987d001009dc846a00d"
Sep 17 18:29:44.632953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:44.632915    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:45.491407 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:45.491372    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:46.290883 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:46.290843    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:46.850511 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:46.850481    2589 scope.go:117] "RemoveContainer" containerID="35629e918ad48436ab6ed4fbfae6b7516e6c5a81f51b328d708053cf9e3fc3f0"
Sep 17 18:29:46.858791 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:46.858772    2589 scope.go:117] "RemoveContainer" containerID="0db121c02a29929c46134925f46db627cb6a596bbec9c4d671f4db49e83e1cd2"
Sep 17 18:29:47.074018 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:47.073977    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:47.307484 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:47.307456    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:29:48.253465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:48.253429    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:50.043853 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:50.043820    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:53.893792 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:53.893752    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:53.968229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:53.968199    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.069029 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:54.068973    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.169453 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:54.169357    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.464623 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:54.464542    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:54.628452 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:54.628415    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.247531 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.247497    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.275164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.275134    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"]
Sep 17 18:29:55.275383 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.275358    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv" podUID="1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0" containerName="payload-processing" containerID="cri-o://3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be" gracePeriod=30
Sep 17 18:29:55.286215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.286178    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"]
Sep 17 18:29:55.286556 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.286523    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb" podUID="00a3a067-e994-4932-83ca-88a14d3e9054" containerName="payload-pre-processing" containerID="cri-o://09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a" gracePeriod=30
Sep 17 18:29:55.315439 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.315410    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.384517 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.384489    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.478175 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.478138    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"]
Sep 17 18:29:55.480146 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.480120    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.483359 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.483334    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4faba0-5wt6r"]
Sep 17 18:29:55.539904 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.539742    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.570270 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.570245    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:29:55.573352 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.573334    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:29:55.683486 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.683445    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7glx\" (UniqueName: \"kubernetes.io/projected/00a3a067-e994-4932-83ca-88a14d3e9054-kube-api-access-n7glx\") pod \"00a3a067-e994-4932-83ca-88a14d3e9054\" (UID: \"00a3a067-e994-4932-83ca-88a14d3e9054\") "
Sep 17 18:29:55.683679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.683552    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn5tv\" (UniqueName: \"kubernetes.io/projected/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-kube-api-access-hn5tv\") pod \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\" (UID: \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\") "
Sep 17 18:29:55.683679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.683623    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a3a067-e994-4932-83ca-88a14d3e9054-plugins-config-volume\") pod \"00a3a067-e994-4932-83ca-88a14d3e9054\" (UID: \"00a3a067-e994-4932-83ca-88a14d3e9054\") "
Sep 17 18:29:55.683818 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.683716    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-plugins-config-volume\") pod \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\" (UID: \"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0\") "
Sep 17 18:29:55.684133 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.684091    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a3a067-e994-4932-83ca-88a14d3e9054-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00a3a067-e994-4932-83ca-88a14d3e9054" (UID: "00a3a067-e994-4932-83ca-88a14d3e9054"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:55.684278 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.684139    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0" (UID: "1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:29:55.686545 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.686513    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a3a067-e994-4932-83ca-88a14d3e9054-kube-api-access-n7glx" (OuterVolumeSpecName: "kube-api-access-n7glx") pod "00a3a067-e994-4932-83ca-88a14d3e9054" (UID: "00a3a067-e994-4932-83ca-88a14d3e9054"). InnerVolumeSpecName "kube-api-access-n7glx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:55.686767 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.686740    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-kube-api-access-hn5tv" (OuterVolumeSpecName: "kube-api-access-hn5tv") pod "1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0" (UID: "1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0"). InnerVolumeSpecName "kube-api-access-hn5tv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:55.726754 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.726722    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:55.785122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.785090    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.785122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.785121    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7glx\" (UniqueName: \"kubernetes.io/projected/00a3a067-e994-4932-83ca-88a14d3e9054-kube-api-access-n7glx\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.785329 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.785138    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn5tv\" (UniqueName: \"kubernetes.io/projected/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0-kube-api-access-hn5tv\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.785329 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.785151    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00a3a067-e994-4932-83ca-88a14d3e9054-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:55.829607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:55.829511    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:56.117451 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.117360    2589 generic.go:356] "Generic (PLEG): container finished" podID="00a3a067-e994-4932-83ca-88a14d3e9054" containerID="09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a" exitCode=0
Sep 17 18:29:56.117631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.117454    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"
Sep 17 18:29:56.117631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.117491    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb" event={"ID":"00a3a067-e994-4932-83ca-88a14d3e9054","Type":"ContainerDied","Data":"09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a"}
Sep 17 18:29:56.117631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.117540    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb" event={"ID":"00a3a067-e994-4932-83ca-88a14d3e9054","Type":"ContainerDied","Data":"467f959cf98895f0aab761fd995acfe80b5e19846577949250b01e7603592dbc"}
Sep 17 18:29:56.117631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.117562    2589 scope.go:117] "RemoveContainer" containerID="09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a"
Sep 17 18:29:56.118855 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.118837    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"
Sep 17 18:29:56.118955 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.118857    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv" event={"ID":"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0","Type":"ContainerDied","Data":"3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be"}
Sep 17 18:29:56.118955 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.118835    2589 generic.go:356] "Generic (PLEG): container finished" podID="1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0" containerID="3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be" exitCode=0
Sep 17 18:29:56.119098 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.118987    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv" event={"ID":"1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0","Type":"ContainerDied","Data":"1ddf8c4236534d687e1fc2be93fe2c24fa961c7a67cd39a19d0d2e64ca42bdc3"}
Sep 17 18:29:56.127287 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.127270    2589 scope.go:117] "RemoveContainer" containerID="09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a"
Sep 17 18:29:56.127570 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:29:56.127551    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a\": container with ID starting with 09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a not found: ID does not exist" containerID="09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a"
Sep 17 18:29:56.127614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.127579    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a"} err="failed to get container status \"09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a\": rpc error: code = NotFound desc = could not find container \"09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a\": container with ID starting with 09a7b03165ce2dd380f0edc36eeb975bc94eda2693626cbed4db2ccbade4997a not found: ID does not exist"
Sep 17 18:29:56.127614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.127596    2589 scope.go:117] "RemoveContainer" containerID="3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be"
Sep 17 18:29:56.135773 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.135756    2589 scope.go:117] "RemoveContainer" containerID="3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be"
Sep 17 18:29:56.136052 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:29:56.136028    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be\": container with ID starting with 3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be not found: ID does not exist" containerID="3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be"
Sep 17 18:29:56.136127 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.136060    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be"} err="failed to get container status \"3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be\": rpc error: code = NotFound desc = could not find container \"3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be\": container with ID starting with 3a355e749ef754ad2b225d625bc5965875729529927dbc962b0a1e05294335be not found: ID does not exist"
Sep 17 18:29:56.141804 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.141783    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"]
Sep 17 18:29:56.147101 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.147082    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4faba0-78d9bc56d6-7msrb"]
Sep 17 18:29:56.156834 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.156816    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"]
Sep 17 18:29:56.165965 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.165946    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4faba0-9f944b5ff-swnmv"]
Sep 17 18:29:56.530247 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.530190    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:56.683926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.683893    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00a3a067-e994-4932-83ca-88a14d3e9054" path="/var/lib/kubelet/pods/00a3a067-e994-4932-83ca-88a14d3e9054/volumes"
Sep 17 18:29:56.684276 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.684262    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="058bac6f-7fb2-4da0-8c74-9bf6160782b4" path="/var/lib/kubelet/pods/058bac6f-7fb2-4da0-8c74-9bf6160782b4/volumes"
Sep 17 18:29:56.684576 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:56.684564    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0" path="/var/lib/kubelet/pods/1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0/volumes"
Sep 17 18:29:57.430059 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:57.429994    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:57.969674 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:57.969632    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7" podUID="04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" containerName="main" containerID="cri-o://62c870347f922b55b947b60eb4e7789a25516f73462757f660e12cabf4d51937" gracePeriod=45
Sep 17 18:29:58.002508 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.002468    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m" podUID="7b08501b-8774-407d-ae48-32db6c7ca4c0" containerName="main" containerID="cri-o://20cc916f56811deb3208563afddd102db2b19e61ba8eed6be154d0f10626093b" gracePeriod=45
Sep 17 18:29:58.133359 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.133328    2589 generic.go:356] "Generic (PLEG): container finished" podID="7b08501b-8774-407d-ae48-32db6c7ca4c0" containerID="20cc916f56811deb3208563afddd102db2b19e61ba8eed6be154d0f10626093b" exitCode=0
Sep 17 18:29:58.133518 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.133397    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m" event={"ID":"7b08501b-8774-407d-ae48-32db6c7ca4c0","Type":"ContainerDied","Data":"20cc916f56811deb3208563afddd102db2b19e61ba8eed6be154d0f10626093b"}
Sep 17 18:29:58.135207 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.135182    2589 generic.go:356] "Generic (PLEG): container finished" podID="04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" containerID="62c870347f922b55b947b60eb4e7789a25516f73462757f660e12cabf4d51937" exitCode=0
Sep 17 18:29:58.135328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.135211    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7" event={"ID":"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a","Type":"ContainerDied","Data":"62c870347f922b55b947b60eb4e7789a25516f73462757f660e12cabf4d51937"}
Sep 17 18:29:58.152041 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.151985    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:58.271765 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.271740    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:29:58.275080 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.275060    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:29:58.408226 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408196    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b08501b-8774-407d-ae48-32db6c7ca4c0-tls-certs\") pod \"7b08501b-8774-407d-ae48-32db6c7ca4c0\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") "
Sep 17 18:29:58.408388 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408240    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-model-cache\") pod \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") "
Sep 17 18:29:58.408388 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408282    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-home\") pod \"7b08501b-8774-407d-ae48-32db6c7ca4c0\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") "
Sep 17 18:29:58.408388 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408317    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-dshm\") pod \"7b08501b-8774-407d-ae48-32db6c7ca4c0\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") "
Sep 17 18:29:58.408388 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408339    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-tmp-dir\") pod \"7b08501b-8774-407d-ae48-32db6c7ca4c0\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") "
Sep 17 18:29:58.408388 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408355    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-dshm\") pod \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") "
Sep 17 18:29:58.408659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408390    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfrk4\" (UniqueName: \"kubernetes.io/projected/7b08501b-8774-407d-ae48-32db6c7ca4c0-kube-api-access-nfrk4\") pod \"7b08501b-8774-407d-ae48-32db6c7ca4c0\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") "
Sep 17 18:29:58.408659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408441    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tls-certs\") pod \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") "
Sep 17 18:29:58.408659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408478    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kcspz\" (UniqueName: \"kubernetes.io/projected/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-kube-api-access-kcspz\") pod \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") "
Sep 17 18:29:58.408659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408512    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-home\") pod \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") "
Sep 17 18:29:58.408659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408539    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tmp-dir\") pod \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\" (UID: \"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a\") "
Sep 17 18:29:58.408659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408605    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-model-cache\") pod \"7b08501b-8774-407d-ae48-32db6c7ca4c0\" (UID: \"7b08501b-8774-407d-ae48-32db6c7ca4c0\") "
Sep 17 18:29:58.408659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408603    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-model-cache" (OuterVolumeSpecName: "model-cache") pod "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" (UID: "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:58.408659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408619    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-home" (OuterVolumeSpecName: "home") pod "7b08501b-8774-407d-ae48-32db6c7ca4c0" (UID: "7b08501b-8774-407d-ae48-32db6c7ca4c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:58.408659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408635    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7b08501b-8774-407d-ae48-32db6c7ca4c0" (UID: "7b08501b-8774-407d-ae48-32db6c7ca4c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:58.409168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.408860    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-home" (OuterVolumeSpecName: "home") pod "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" (UID: "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:58.409168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.409098    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "7b08501b-8774-407d-ae48-32db6c7ca4c0" (UID: "7b08501b-8774-407d-ae48-32db6c7ca4c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:58.409283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.409198    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.409283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.409234    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.409283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.409247    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.409283 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.409262    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.409656 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.409446    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" (UID: "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:58.410787 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.410764    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b08501b-8774-407d-ae48-32db6c7ca4c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7b08501b-8774-407d-ae48-32db6c7ca4c0" (UID: "7b08501b-8774-407d-ae48-32db6c7ca4c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:58.410971 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.410940    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" (UID: "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:29:58.411246 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.411219    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-dshm" (OuterVolumeSpecName: "dshm") pod "7b08501b-8774-407d-ae48-32db6c7ca4c0" (UID: "7b08501b-8774-407d-ae48-32db6c7ca4c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:58.411378 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.411359    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-kube-api-access-kcspz" (OuterVolumeSpecName: "kube-api-access-kcspz") pod "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" (UID: "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a"). InnerVolumeSpecName "kube-api-access-kcspz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:58.411571 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.411544    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-dshm" (OuterVolumeSpecName: "dshm") pod "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" (UID: "04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:29:58.411875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.411850    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b08501b-8774-407d-ae48-32db6c7ca4c0-kube-api-access-nfrk4" (OuterVolumeSpecName: "kube-api-access-nfrk4") pod "7b08501b-8774-407d-ae48-32db6c7ca4c0" (UID: "7b08501b-8774-407d-ae48-32db6c7ca4c0"). InnerVolumeSpecName "kube-api-access-nfrk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:29:58.509834 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.509802    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.509834 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.509827    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcspz\" (UniqueName: \"kubernetes.io/projected/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-kube-api-access-kcspz\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.509834 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.509837    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.510079 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.509847    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.510079 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.509855    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b08501b-8774-407d-ae48-32db6c7ca4c0-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.510079 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.509862    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b08501b-8774-407d-ae48-32db6c7ca4c0-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.510079 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.509869    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.510079 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.509876    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfrk4\" (UniqueName: \"kubernetes.io/projected/7b08501b-8774-407d-ae48-32db6c7ca4c0-kube-api-access-nfrk4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:29:58.949400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:58.949310    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:29:59.142177 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.142142    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"
Sep 17 18:29:59.142616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.142192    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7" event={"ID":"04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a","Type":"ContainerDied","Data":"6abc9df37adbc73045d50d471d5bb0a1200d7eb423952225958bce0b8d574b1e"}
Sep 17 18:29:59.142616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.142251    2589 scope.go:117] "RemoveContainer" containerID="62c870347f922b55b947b60eb4e7789a25516f73462757f660e12cabf4d51937"
Sep 17 18:29:59.143709 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.143689    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m" event={"ID":"7b08501b-8774-407d-ae48-32db6c7ca4c0","Type":"ContainerDied","Data":"5f1e0818b181433eb29603053296536c5f93036e9be36439567902abc494cc75"}
Sep 17 18:29:59.143790 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.143706    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"
Sep 17 18:29:59.151379 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.151357    2589 scope.go:117] "RemoveContainer" containerID="20cc916f56811deb3208563afddd102db2b19e61ba8eed6be154d0f10626093b"
Sep 17 18:29:59.162468 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.162446    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"]
Sep 17 18:29:59.171626 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.171588    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/simulator-w4-b18433-kserve-855849ffb4-5lq9m"]
Sep 17 18:29:59.181070 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.181047    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"]
Sep 17 18:29:59.185372 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.185347    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-b18433/distinct-2-w4-b18433-kserve-5b5d585f87-pszb7"]
Sep 17 18:29:59.672504 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:29:59.672468    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:00.151767 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.151741    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf"]
Sep 17 18:30:00.152155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152119    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00a3a067-e994-4932-83ca-88a14d3e9054" containerName="payload-pre-processing"
Sep 17 18:30:00.152155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152132    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="00a3a067-e994-4932-83ca-88a14d3e9054" containerName="payload-pre-processing"
Sep 17 18:30:00.152155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152147    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="058bac6f-7fb2-4da0-8c74-9bf6160782b4" containerName="revoke-keys"
Sep 17 18:30:00.152155 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152153    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="058bac6f-7fb2-4da0-8c74-9bf6160782b4" containerName="revoke-keys"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152163    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0" containerName="payload-processing"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152169    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0" containerName="payload-processing"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152181    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b08501b-8774-407d-ae48-32db6c7ca4c0" containerName="main"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152186    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b08501b-8774-407d-ae48-32db6c7ca4c0" containerName="main"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152191    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" containerName="main"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152196    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" containerName="main"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152251    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" containerName="main"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152257    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="058bac6f-7fb2-4da0-8c74-9bf6160782b4" containerName="revoke-keys"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152266    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b08501b-8774-407d-ae48-32db6c7ca4c0" containerName="main"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152270    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="00a3a067-e994-4932-83ca-88a14d3e9054" containerName="payload-pre-processing"
Sep 17 18:30:00.152295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.152275    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bf34e9c-af10-4b36-af49-aa0b9d3a9ff0" containerName="payload-processing"
Sep 17 18:30:00.156953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.156935    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx"]
Sep 17 18:30:00.157069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.157053    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf"
Sep 17 18:30:00.159852 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.159831    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:30:00.160173 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.160159    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf"]
Sep 17 18:30:00.160269 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.160257    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx"
Sep 17 18:30:00.163227 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.163210    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf"
Sep 17 18:30:00.167747 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.167731    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf"
Sep 17 18:30:00.171474 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.171165    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx"]
Sep 17 18:30:00.173718 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.173697    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf"]
Sep 17 18:30:00.178201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.177935    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf"]
Sep 17 18:30:00.178201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.177993    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx"
Sep 17 18:30:00.188163 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.188138    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl"]
Sep 17 18:30:00.192647 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.192622    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl"
Sep 17 18:30:00.209614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.209536    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl"]
Sep 17 18:30:00.224518 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.224495    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf"
Sep 17 18:30:00.238371 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.238348    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl"
Sep 17 18:30:00.458406 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.458382    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf"]
Sep 17 18:30:00.461025 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:30:00.460982    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1e2af21_c859_415e_a596_a24f80f46ac0.slice/crio-c4437cfb4a047504b580982afdce28716d28f50906c0e132483400e664fec351 WatchSource:0}: Error finding container c4437cfb4a047504b580982afdce28716d28f50906c0e132483400e664fec351: Status 404 returned error can't find the container with id c4437cfb4a047504b580982afdce28716d28f50906c0e132483400e664fec351
Sep 17 18:30:00.476151 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.476129    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx"]
Sep 17 18:30:00.477567 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:30:00.477539    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ce5059_1fbb_4871_aa0a_0133da21c633.slice/crio-fee39fd8d5b7cd041aa22469742b2244478e990d685c74053e1df572da891c2c WatchSource:0}: Error finding container fee39fd8d5b7cd041aa22469742b2244478e990d685c74053e1df572da891c2c: Status 404 returned error can't find the container with id fee39fd8d5b7cd041aa22469742b2244478e990d685c74053e1df572da891c2c
Sep 17 18:30:00.529235 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.529207    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl"]
Sep 17 18:30:00.532691 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:30:00.532665    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbabc825_f2e7_4ee7_80ec_207341817f7c.slice/crio-de2446a65e23155a09375b8cb7fe4436832eed49173487358e5f453800198b0b WatchSource:0}: Error finding container de2446a65e23155a09375b8cb7fe4436832eed49173487358e5f453800198b0b: Status 404 returned error can't find the container with id de2446a65e23155a09375b8cb7fe4436832eed49173487358e5f453800198b0b
Sep 17 18:30:00.538358 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.538333    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf"]
Sep 17 18:30:00.540280 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:30:00.540259    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f413b49_e915_44ca_a269_10b219749a68.slice/crio-652aac8169b7ba675efa8c147474f3a70cc0daaf3e71b753d010780fd009808c WatchSource:0}: Error finding container 652aac8169b7ba675efa8c147474f3a70cc0daaf3e71b753d010780fd009808c: Status 404 returned error can't find the container with id 652aac8169b7ba675efa8c147474f3a70cc0daaf3e71b753d010780fd009808c
Sep 17 18:30:00.564742 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.564718    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:00.685532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.685502    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a" path="/var/lib/kubelet/pods/04dc3bf5-a77f-4790-ab6c-be5cd0a8b70a/volumes"
Sep 17 18:30:00.685916 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:00.685903    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b08501b-8774-407d-ae48-32db6c7ca4c0" path="/var/lib/kubelet/pods/7b08501b-8774-407d-ae48-32db6c7ca4c0/volumes"
Sep 17 18:30:01.158792 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.158689    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf" event={"ID":"2f413b49-e915-44ca-a269-10b219749a68","Type":"ContainerStarted","Data":"d7614fa1148a67b452bc5ee26a9626fc4d491ff76b83fd4e04707a7237e32743"}
Sep 17 18:30:01.158792 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.158749    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf" event={"ID":"2f413b49-e915-44ca-a269-10b219749a68","Type":"ContainerStarted","Data":"652aac8169b7ba675efa8c147474f3a70cc0daaf3e71b753d010780fd009808c"}
Sep 17 18:30:01.160351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.160319    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl" event={"ID":"cbabc825-f2e7-4ee7-80ec-207341817f7c","Type":"ContainerStarted","Data":"37a58b74c3e5b459e9d0eff5320cccd78b4e096fcd1534b4cf60c510fad40f0f"}
Sep 17 18:30:01.160472 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.160366    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl" event={"ID":"cbabc825-f2e7-4ee7-80ec-207341817f7c","Type":"ContainerStarted","Data":"de2446a65e23155a09375b8cb7fe4436832eed49173487358e5f453800198b0b"}
Sep 17 18:30:01.161898 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.161876    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx" event={"ID":"08ce5059-1fbb-4871-aa0a-0133da21c633","Type":"ContainerStarted","Data":"ce06a6048f2e76b5df5a95402539bbe25f3001652baf8bc1570868f16c896aae"}
Sep 17 18:30:01.162037 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.161923    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx" event={"ID":"08ce5059-1fbb-4871-aa0a-0133da21c633","Type":"ContainerStarted","Data":"fee39fd8d5b7cd041aa22469742b2244478e990d685c74053e1df572da891c2c"}
Sep 17 18:30:01.163516 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.163496    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf" event={"ID":"a1e2af21-c859-415e-a596-a24f80f46ac0","Type":"ContainerStarted","Data":"80ef95599fb582471063d9835690f3ed3b886d38b51197028794e6d8d7b37e39"}
Sep 17 18:30:01.163616 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.163546    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf" event={"ID":"a1e2af21-c859-415e-a596-a24f80f46ac0","Type":"ContainerStarted","Data":"c4437cfb4a047504b580982afdce28716d28f50906c0e132483400e664fec351"}
Sep 17 18:30:01.174352 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.174310    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf" podStartSLOduration=1.174295173 podStartE2EDuration="1.174295173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.173495267 +0000 UTC m=+1636.853181123" watchObservedRunningTime="2026-09-17 18:30:01.174295173 +0000 UTC m=+1636.853981027"
Sep 17 18:30:01.190800 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.190750    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf" podStartSLOduration=1.190733961 podStartE2EDuration="1.190733961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.187887453 +0000 UTC m=+1636.867573321" watchObservedRunningTime="2026-09-17 18:30:01.190733961 +0000 UTC m=+1636.870419851"
Sep 17 18:30:01.203674 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.203629    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl" podStartSLOduration=1.203615465 podStartE2EDuration="1.203615465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.202534934 +0000 UTC m=+1636.882220790" watchObservedRunningTime="2026-09-17 18:30:01.203615465 +0000 UTC m=+1636.883301318"
Sep 17 18:30:01.219108 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.219064    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx" podStartSLOduration=1.219050046 podStartE2EDuration="1.219050046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:30:01.217707633 +0000 UTC m=+1636.897393488" watchObservedRunningTime="2026-09-17 18:30:01.219050046 +0000 UTC m=+1636.898735896"
Sep 17 18:30:01.372525 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:01.372485    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.170982 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.170950    2589 generic.go:356] "Generic (PLEG): container finished" podID="2f413b49-e915-44ca-a269-10b219749a68" containerID="d7614fa1148a67b452bc5ee26a9626fc4d491ff76b83fd4e04707a7237e32743" exitCode=0
Sep 17 18:30:02.171412 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.171042    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf" event={"ID":"2f413b49-e915-44ca-a269-10b219749a68","Type":"ContainerDied","Data":"d7614fa1148a67b452bc5ee26a9626fc4d491ff76b83fd4e04707a7237e32743"}
Sep 17 18:30:02.172610 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.172585    2589 generic.go:356] "Generic (PLEG): container finished" podID="cbabc825-f2e7-4ee7-80ec-207341817f7c" containerID="37a58b74c3e5b459e9d0eff5320cccd78b4e096fcd1534b4cf60c510fad40f0f" exitCode=0
Sep 17 18:30:02.172755 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.172661    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl" event={"ID":"cbabc825-f2e7-4ee7-80ec-207341817f7c","Type":"ContainerDied","Data":"37a58b74c3e5b459e9d0eff5320cccd78b4e096fcd1534b4cf60c510fad40f0f"}
Sep 17 18:30:02.174178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.174160    2589 generic.go:356] "Generic (PLEG): container finished" podID="08ce5059-1fbb-4871-aa0a-0133da21c633" containerID="ce06a6048f2e76b5df5a95402539bbe25f3001652baf8bc1570868f16c896aae" exitCode=0
Sep 17 18:30:02.174297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.174231    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx" event={"ID":"08ce5059-1fbb-4871-aa0a-0133da21c633","Type":"ContainerDied","Data":"ce06a6048f2e76b5df5a95402539bbe25f3001652baf8bc1570868f16c896aae"}
Sep 17 18:30:02.175744 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.175725    2589 generic.go:356] "Generic (PLEG): container finished" podID="a1e2af21-c859-415e-a596-a24f80f46ac0" containerID="80ef95599fb582471063d9835690f3ed3b886d38b51197028794e6d8d7b37e39" exitCode=0
Sep 17 18:30:02.175819 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.175760    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf" event={"ID":"a1e2af21-c859-415e-a596-a24f80f46ac0","Type":"ContainerDied","Data":"80ef95599fb582471063d9835690f3ed3b886d38b51197028794e6d8d7b37e39"}
Sep 17 18:30:02.197335 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.197307    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.510755 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.510726    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:02.591210 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:02.591163    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:03.019610 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.019566    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:03.122996 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.122941    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:03.265252 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.265223    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"]
Sep 17 18:30:03.265674 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.265410    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy" containerID="cri-o://8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43" gracePeriod=30
Sep 17 18:30:03.307134 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.307057    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:03.371850 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.370832    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl"
Sep 17 18:30:03.383869 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.379338    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf"
Sep 17 18:30:03.411919 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.411899    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf"
Sep 17 18:30:03.415722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.415702    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx"
Sep 17 18:30:03.628937 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.628855    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"]
Sep 17 18:30:03.629247 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629234    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08ce5059-1fbb-4871-aa0a-0133da21c633" containerName="cleanup"
Sep 17 18:30:03.629287 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629249    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="08ce5059-1fbb-4871-aa0a-0133da21c633" containerName="cleanup"
Sep 17 18:30:03.629287 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629259    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1e2af21-c859-415e-a596-a24f80f46ac0" containerName="cleanup"
Sep 17 18:30:03.629287 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629265    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1e2af21-c859-415e-a596-a24f80f46ac0" containerName="cleanup"
Sep 17 18:30:03.629287 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629276    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f413b49-e915-44ca-a269-10b219749a68" containerName="cleanup"
Sep 17 18:30:03.629287 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629282    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f413b49-e915-44ca-a269-10b219749a68" containerName="cleanup"
Sep 17 18:30:03.629447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629297    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbabc825-f2e7-4ee7-80ec-207341817f7c" containerName="cleanup"
Sep 17 18:30:03.629447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629302    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbabc825-f2e7-4ee7-80ec-207341817f7c" containerName="cleanup"
Sep 17 18:30:03.629447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629362    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbabc825-f2e7-4ee7-80ec-207341817f7c" containerName="cleanup"
Sep 17 18:30:03.629447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629372    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f413b49-e915-44ca-a269-10b219749a68" containerName="cleanup"
Sep 17 18:30:03.629447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629379    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1e2af21-c859-415e-a596-a24f80f46ac0" containerName="cleanup"
Sep 17 18:30:03.629447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.629384    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="08ce5059-1fbb-4871-aa0a-0133da21c633" containerName="cleanup"
Sep 17 18:30:03.632312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.632298    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"
Sep 17 18:30:03.634873 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.634847    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:30:03.642329 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.642306    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"]
Sep 17 18:30:03.761716 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.761674    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef1731-75f3-43de-ac55-167e7c68765b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b06908-nc4mh\" (UID: \"69ef1731-75f3-43de-ac55-167e7c68765b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"
Sep 17 18:30:03.863075 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.863040    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef1731-75f3-43de-ac55-167e7c68765b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b06908-nc4mh\" (UID: \"69ef1731-75f3-43de-ac55-167e7c68765b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"
Sep 17 18:30:03.863621 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.863601    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef1731-75f3-43de-ac55-167e7c68765b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-b06908-nc4mh\" (UID: \"69ef1731-75f3-43de-ac55-167e7c68765b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"
Sep 17 18:30:03.944162 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:03.944084    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"
Sep 17 18:30:04.186304 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.186279    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf"
Sep 17 18:30:04.186504 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.186321    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf" event={"ID":"2f413b49-e915-44ca-a269-10b219749a68","Type":"ContainerDied","Data":"652aac8169b7ba675efa8c147474f3a70cc0daaf3e71b753d010780fd009808c"}
Sep 17 18:30:04.186504 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.186367    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="652aac8169b7ba675efa8c147474f3a70cc0daaf3e71b753d010780fd009808c"
Sep 17 18:30:04.187733 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.187716    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl"
Sep 17 18:30:04.187858 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.187718    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29827830-bv6xl" event={"ID":"cbabc825-f2e7-4ee7-80ec-207341817f7c","Type":"ContainerDied","Data":"de2446a65e23155a09375b8cb7fe4436832eed49173487358e5f453800198b0b"}
Sep 17 18:30:04.187921 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.187848    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de2446a65e23155a09375b8cb7fe4436832eed49173487358e5f453800198b0b"
Sep 17 18:30:04.189070 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.189047    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx" event={"ID":"08ce5059-1fbb-4871-aa0a-0133da21c633","Type":"ContainerDied","Data":"fee39fd8d5b7cd041aa22469742b2244478e990d685c74053e1df572da891c2c"}
Sep 17 18:30:04.189162 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.189093    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fee39fd8d5b7cd041aa22469742b2244478e990d685c74053e1df572da891c2c"
Sep 17 18:30:04.189162 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.189055    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx"
Sep 17 18:30:04.190462 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.190428    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf" event={"ID":"a1e2af21-c859-415e-a596-a24f80f46ac0","Type":"ContainerDied","Data":"c4437cfb4a047504b580982afdce28716d28f50906c0e132483400e664fec351"}
Sep 17 18:30:04.190547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.190482    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4437cfb4a047504b580982afdce28716d28f50906c0e132483400e664fec351"
Sep 17 18:30:04.190547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.190434    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf"
Sep 17 18:30:04.220521 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.220470    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"]
Sep 17 18:30:04.222695 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:04.222668    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:04.224747 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:30:04.224722    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ef1731_75f3_43de_ac55_167e7c68765b.slice/crio-a63f6c7ca0c542e9c088b80ecdbc0016779be9f02f852efe966c5ae59f5c0af2 WatchSource:0}: Error finding container a63f6c7ca0c542e9c088b80ecdbc0016779be9f02f852efe966c5ae59f5c0af2: Status 404 returned error can't find the container with id a63f6c7ca0c542e9c088b80ecdbc0016779be9f02f852efe966c5ae59f5c0af2
Sep 17 18:30:05.037973 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:05.037941    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:05.196675 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:05.196583    2589 generic.go:356] "Generic (PLEG): container finished" podID="69ef1731-75f3-43de-ac55-167e7c68765b" containerID="fc4899852e5acbae93a9b31fe547c7d27b0a3897022d69df62f072ba1ba77f53" exitCode=0
Sep 17 18:30:05.196675 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:05.196657    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh" event={"ID":"69ef1731-75f3-43de-ac55-167e7c68765b","Type":"ContainerDied","Data":"fc4899852e5acbae93a9b31fe547c7d27b0a3897022d69df62f072ba1ba77f53"}
Sep 17 18:30:05.196896 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:05.196704    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh" event={"ID":"69ef1731-75f3-43de-ac55-167e7c68765b","Type":"ContainerStarted","Data":"a63f6c7ca0c542e9c088b80ecdbc0016779be9f02f852efe966c5ae59f5c0af2"}
Sep 17 18:30:05.742932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:05.742893    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.335152 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:06.335127    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"
Sep 17 18:30:06.450998 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:06.450962    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:06.489679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:06.489601    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef1731-75f3-43de-ac55-167e7c68765b-maas-api-service-ca\") pod \"69ef1731-75f3-43de-ac55-167e7c68765b\" (UID: \"69ef1731-75f3-43de-ac55-167e7c68765b\") "
Sep 17 18:30:06.489912 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:06.489891    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ef1731-75f3-43de-ac55-167e7c68765b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69ef1731-75f3-43de-ac55-167e7c68765b" (UID: "69ef1731-75f3-43de-ac55-167e7c68765b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:06.591106 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:06.591069    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69ef1731-75f3-43de-ac55-167e7c68765b-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:07.210336 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:07.210306    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"
Sep 17 18:30:07.210518 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:07.210339    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh" event={"ID":"69ef1731-75f3-43de-ac55-167e7c68765b","Type":"ContainerDied","Data":"a63f6c7ca0c542e9c088b80ecdbc0016779be9f02f852efe966c5ae59f5c0af2"}
Sep 17 18:30:07.210518 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:07.210387    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a63f6c7ca0c542e9c088b80ecdbc0016779be9f02f852efe966c5ae59f5c0af2"
Sep 17 18:30:07.342757 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:07.342722    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.113149 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.113091    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:08.347540 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.347511    2589 patch_prober.go:28] interesting pod/e2e-api-a-4faba0-openshift-default-55b947b884-8h457 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused" start-of-body=
Sep 17 18:30:08.347811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.347568    2589 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.69:15021/healthz/ready\": dial tcp 10.132.0.69:15021: connect: connection refused"
Sep 17 18:30:08.522024 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.521981    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:30:08.609078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609044    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33871cf8-346a-47be-9aee-21d073db84d0-istiod-ca-cert\") pod \"33871cf8-346a-47be-9aee-21d073db84d0\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") "
Sep 17 18:30:08.609245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609095    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-data\") pod \"33871cf8-346a-47be-9aee-21d073db84d0\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") "
Sep 17 18:30:08.609245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609121    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-certs\") pod \"33871cf8-346a-47be-9aee-21d073db84d0\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") "
Sep 17 18:30:08.609245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609147    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-credential-socket\") pod \"33871cf8-346a-47be-9aee-21d073db84d0\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") "
Sep 17 18:30:08.609245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609182    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-envoy\") pod \"33871cf8-346a-47be-9aee-21d073db84d0\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") "
Sep 17 18:30:08.609245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609233    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33871cf8-346a-47be-9aee-21d073db84d0-istio-podinfo\") pod \"33871cf8-346a-47be-9aee-21d073db84d0\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") "
Sep 17 18:30:08.609573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609273    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-istio-token\") pod \"33871cf8-346a-47be-9aee-21d073db84d0\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") "
Sep 17 18:30:08.609573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609333    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4s8s7\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-kube-api-access-4s8s7\") pod \"33871cf8-346a-47be-9aee-21d073db84d0\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") "
Sep 17 18:30:08.609573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609418    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-socket\") pod \"33871cf8-346a-47be-9aee-21d073db84d0\" (UID: \"33871cf8-346a-47be-9aee-21d073db84d0\") "
Sep 17 18:30:08.609573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609452    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "33871cf8-346a-47be-9aee-21d073db84d0" (UID: "33871cf8-346a-47be-9aee-21d073db84d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:08.609573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609463    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-data" (OuterVolumeSpecName: "istio-data") pod "33871cf8-346a-47be-9aee-21d073db84d0" (UID: "33871cf8-346a-47be-9aee-21d073db84d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:08.609573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609478    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "33871cf8-346a-47be-9aee-21d073db84d0" (UID: "33871cf8-346a-47be-9aee-21d073db84d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:08.609573 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609497    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33871cf8-346a-47be-9aee-21d073db84d0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "33871cf8-346a-47be-9aee-21d073db84d0" (UID: "33871cf8-346a-47be-9aee-21d073db84d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:08.609877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609722    2589 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33871cf8-346a-47be-9aee-21d073db84d0-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:08.609877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609739    2589 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:08.609877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609752    2589 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:08.609877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609764    2589 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:08.609877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.609771    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "33871cf8-346a-47be-9aee-21d073db84d0" (UID: "33871cf8-346a-47be-9aee-21d073db84d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:08.611713 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.611685    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-istio-token" (OuterVolumeSpecName: "istio-token") pod "33871cf8-346a-47be-9aee-21d073db84d0" (UID: "33871cf8-346a-47be-9aee-21d073db84d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:08.611828 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.611794    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33871cf8-346a-47be-9aee-21d073db84d0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "33871cf8-346a-47be-9aee-21d073db84d0" (UID: "33871cf8-346a-47be-9aee-21d073db84d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:30:08.612081 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.612058    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "33871cf8-346a-47be-9aee-21d073db84d0" (UID: "33871cf8-346a-47be-9aee-21d073db84d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:08.612148 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.612094    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-kube-api-access-4s8s7" (OuterVolumeSpecName: "kube-api-access-4s8s7") pod "33871cf8-346a-47be-9aee-21d073db84d0" (UID: "33871cf8-346a-47be-9aee-21d073db84d0"). InnerVolumeSpecName "kube-api-access-4s8s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:08.711303 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.711267    2589 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33871cf8-346a-47be-9aee-21d073db84d0-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:08.711303 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.711302    2589 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:08.711466 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.711318    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s8s7\" (UniqueName: \"kubernetes.io/projected/33871cf8-346a-47be-9aee-21d073db84d0-kube-api-access-4s8s7\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:08.711466 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.711333    2589 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:08.711466 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.711347    2589 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33871cf8-346a-47be-9aee-21d073db84d0-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:08.904776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:08.904735    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:09.221471 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.221433    2589 generic.go:356] "Generic (PLEG): container finished" podID="33871cf8-346a-47be-9aee-21d073db84d0" containerID="8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43" exitCode=0
Sep 17 18:30:09.221628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.221487    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"
Sep 17 18:30:09.221628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.221516    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" event={"ID":"33871cf8-346a-47be-9aee-21d073db84d0","Type":"ContainerDied","Data":"8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43"}
Sep 17 18:30:09.221628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.221573    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457" event={"ID":"33871cf8-346a-47be-9aee-21d073db84d0","Type":"ContainerDied","Data":"d864afe6bece9acd9f81c09198da31efa4b33ddcd8fab0b94834d5920a04616f"}
Sep 17 18:30:09.221628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.221595    2589 scope.go:117] "RemoveContainer" containerID="8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43"
Sep 17 18:30:09.233452 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.233435    2589 scope.go:117] "RemoveContainer" containerID="6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30"
Sep 17 18:30:09.242282 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.242263    2589 scope.go:117] "RemoveContainer" containerID="8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43"
Sep 17 18:30:09.242403 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.242380    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"]
Sep 17 18:30:09.242603 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:30:09.242579    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43\": container with ID starting with 8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43 not found: ID does not exist" containerID="8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43"
Sep 17 18:30:09.242781 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.242613    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43"} err="failed to get container status \"8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43\": rpc error: code = NotFound desc = could not find container \"8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43\": container with ID starting with 8c9abd8bb710ed7aa999989ce32b2990bbfffef8bfb01b155d6b0bd768b1fb43 not found: ID does not exist"
Sep 17 18:30:09.242781 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.242630    2589 scope.go:117] "RemoveContainer" containerID="6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30"
Sep 17 18:30:09.242895 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:30:09.242879    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30\": container with ID starting with 6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30 not found: ID does not exist" containerID="6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30"
Sep 17 18:30:09.242930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.242901    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30"} err="failed to get container status \"6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30\": rpc error: code = NotFound desc = could not find container \"6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30\": container with ID starting with 6dfd5764f8c792493a9a15c430955a4cc8cc295f95af462c10984a1d24a08a30 not found: ID does not exist"
Sep 17 18:30:09.246447 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.246419    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-4faba0-openshift-default-55b947b884-8h457"]
Sep 17 18:30:09.634213 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:09.634173    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:10.687219 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:10.685277    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33871cf8-346a-47be-9aee-21d073db84d0" path="/var/lib/kubelet/pods/33871cf8-346a-47be-9aee-21d073db84d0/volumes"
Sep 17 18:30:13.321580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.321545    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:13.432229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432197    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"]
Sep 17 18:30:13.432544 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432533    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ef1731-75f3-43de-ac55-167e7c68765b" containerName="revoke-keys"
Sep 17 18:30:13.432589 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432547    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ef1731-75f3-43de-ac55-167e7c68765b" containerName="revoke-keys"
Sep 17 18:30:13.432589 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432561    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy"
Sep 17 18:30:13.432589 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432567    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy"
Sep 17 18:30:13.432589 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432585    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy"
Sep 17 18:30:13.432716 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432591    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy"
Sep 17 18:30:13.432716 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432644    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy"
Sep 17 18:30:13.432716 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432650    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="33871cf8-346a-47be-9aee-21d073db84d0" containerName="istio-proxy"
Sep 17 18:30:13.432716 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.432656    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ef1731-75f3-43de-ac55-167e7c68765b" containerName="revoke-keys"
Sep 17 18:30:13.437214 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.437197    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"
Sep 17 18:30:13.439777 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.439759    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:30:13.439906 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.439793    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:30:13.443564 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.443542    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"]
Sep 17 18:30:13.556954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.556920    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628b1a0f-a488-426e-bd20-0a2ced10af4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp\" (UID: \"628b1a0f-a488-426e-bd20-0a2ced10af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"
Sep 17 18:30:13.658425 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.658354    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628b1a0f-a488-426e-bd20-0a2ced10af4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp\" (UID: \"628b1a0f-a488-426e-bd20-0a2ced10af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"
Sep 17 18:30:13.658957 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.658938    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628b1a0f-a488-426e-bd20-0a2ced10af4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp\" (UID: \"628b1a0f-a488-426e-bd20-0a2ced10af4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"
Sep 17 18:30:13.749948 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.749917    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"
Sep 17 18:30:13.843758 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:13.843710    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:14.021888 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:14.021864    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"]
Sep 17 18:30:14.023976 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:30:14.023951    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628b1a0f_a488_426e_bd20_0a2ced10af4c.slice/crio-73546a7dd719c342284266d1b571d185b8b27ec478ad53a7dbb542f72c2b5b8b WatchSource:0}: Error finding container 73546a7dd719c342284266d1b571d185b8b27ec478ad53a7dbb542f72c2b5b8b: Status 404 returned error can't find the container with id 73546a7dd719c342284266d1b571d185b8b27ec478ad53a7dbb542f72c2b5b8b
Sep 17 18:30:14.251690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:14.251655    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp" event={"ID":"628b1a0f-a488-426e-bd20-0a2ced10af4c","Type":"ContainerStarted","Data":"73546a7dd719c342284266d1b571d185b8b27ec478ad53a7dbb542f72c2b5b8b"}
Sep 17 18:30:14.839113 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:14.839081    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:15.258393 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:15.258353    2589 generic.go:356] "Generic (PLEG): container finished" podID="628b1a0f-a488-426e-bd20-0a2ced10af4c" containerID="aaadf039dd6e0135b8968f450a9e426b5402a0d7fe2ed855468d5209b79c00f7" exitCode=0
Sep 17 18:30:15.258604 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:15.258449    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp" event={"ID":"628b1a0f-a488-426e-bd20-0a2ced10af4c","Type":"ContainerDied","Data":"aaadf039dd6e0135b8968f450a9e426b5402a0d7fe2ed855468d5209b79c00f7"}
Sep 17 18:30:15.571945 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:15.571851    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.331596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:16.331565    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:16.397099 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:16.397075    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"
Sep 17 18:30:16.584181 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:16.584107    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628b1a0f-a488-426e-bd20-0a2ced10af4c-maas-api-service-ca\") pod \"628b1a0f-a488-426e-bd20-0a2ced10af4c\" (UID: \"628b1a0f-a488-426e-bd20-0a2ced10af4c\") "
Sep 17 18:30:16.584440 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:16.584418    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628b1a0f-a488-426e-bd20-0a2ced10af4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "628b1a0f-a488-426e-bd20-0a2ced10af4c" (UID: "628b1a0f-a488-426e-bd20-0a2ced10af4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:16.684856 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:16.684823    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/628b1a0f-a488-426e-bd20-0a2ced10af4c-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:17.219393 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:17.219361    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:17.270184 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:17.270157    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"
Sep 17 18:30:17.270184 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:17.270174    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp" event={"ID":"628b1a0f-a488-426e-bd20-0a2ced10af4c","Type":"ContainerDied","Data":"73546a7dd719c342284266d1b571d185b8b27ec478ad53a7dbb542f72c2b5b8b"}
Sep 17 18:30:17.270393 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:17.270221    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73546a7dd719c342284266d1b571d185b8b27ec478ad53a7dbb542f72c2b5b8b"
Sep 17 18:30:17.899525 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:17.899496    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.323519 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:18.323467    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"]
Sep 17 18:30:18.332869 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:18.332836    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"]
Sep 17 18:30:18.340685 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:18.340655    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"]
Sep 17 18:30:18.399561 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:18.399532    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:18.742368 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:18.742283    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.434030 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:19.433976    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.862307 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:19.862271    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:19.961354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:19.961324    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.306880 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.306851    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.337178 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.337147    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"]
Sep 17 18:30:20.337387 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.337362    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6" podUID="1509c79d-9c57-4cb2-80cc-94a55a0d3103" containerName="payload-processing" containerID="cri-o://94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710" gracePeriod=30
Sep 17 18:30:20.401975 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.401942    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.501184 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.501146    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.515438 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.515407    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"]
Sep 17 18:30:20.518919 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.518879    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-b06908-nc4mh"]
Sep 17 18:30:20.576243 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.576210    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.589902 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.589882    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:30:20.620034 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.619990    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6srvp\" (UniqueName: \"kubernetes.io/projected/1509c79d-9c57-4cb2-80cc-94a55a0d3103-kube-api-access-6srvp\") pod \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\" (UID: \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\") "
Sep 17 18:30:20.620166 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.620150    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1509c79d-9c57-4cb2-80cc-94a55a0d3103-plugins-config-volume\") pod \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\" (UID: \"1509c79d-9c57-4cb2-80cc-94a55a0d3103\") "
Sep 17 18:30:20.621049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.620988    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1509c79d-9c57-4cb2-80cc-94a55a0d3103-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1509c79d-9c57-4cb2-80cc-94a55a0d3103" (UID: "1509c79d-9c57-4cb2-80cc-94a55a0d3103"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:20.629481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.621383    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1509c79d-9c57-4cb2-80cc-94a55a0d3103-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:20.629481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.623776    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1509c79d-9c57-4cb2-80cc-94a55a0d3103-kube-api-access-6srvp" (OuterVolumeSpecName: "kube-api-access-6srvp") pod "1509c79d-9c57-4cb2-80cc-94a55a0d3103" (UID: "1509c79d-9c57-4cb2-80cc-94a55a0d3103"). InnerVolumeSpecName "kube-api-access-6srvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:20.683654 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.683621    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69ef1731-75f3-43de-ac55-167e7c68765b" path="/var/lib/kubelet/pods/69ef1731-75f3-43de-ac55-167e7c68765b/volumes"
Sep 17 18:30:20.722568 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.722538    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6srvp\" (UniqueName: \"kubernetes.io/projected/1509c79d-9c57-4cb2-80cc-94a55a0d3103-kube-api-access-6srvp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:20.866233 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.866138    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:20.968169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:20.968132    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.292228 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.292190    2589 generic.go:356] "Generic (PLEG): container finished" podID="1509c79d-9c57-4cb2-80cc-94a55a0d3103" containerID="94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710" exitCode=0
Sep 17 18:30:21.292424 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.292237    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6" event={"ID":"1509c79d-9c57-4cb2-80cc-94a55a0d3103","Type":"ContainerDied","Data":"94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710"}
Sep 17 18:30:21.292424 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.292255    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"
Sep 17 18:30:21.292424 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.292292    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6" event={"ID":"1509c79d-9c57-4cb2-80cc-94a55a0d3103","Type":"ContainerDied","Data":"0ecb78e7c9cdba06192c42a4c8e58658439640afc8035a078205d6157338db83"}
Sep 17 18:30:21.292424 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.292320    2589 scope.go:117] "RemoveContainer" containerID="94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710"
Sep 17 18:30:21.301619 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.301602    2589 scope.go:117] "RemoveContainer" containerID="94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710"
Sep 17 18:30:21.301880 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:30:21.301862    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710\": container with ID starting with 94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710 not found: ID does not exist" containerID="94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710"
Sep 17 18:30:21.301927 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.301887    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710"} err="failed to get container status \"94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710\": rpc error: code = NotFound desc = could not find container \"94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710\": container with ID starting with 94601e35ae1a5c55a7c2be65b7bce7c486912644de8e6cb8716b418b20bda710 not found: ID does not exist"
Sep 17 18:30:21.316501 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.316475    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"]
Sep 17 18:30:21.320672 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.320653    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-b06908-64c748b995-9qfh6"]
Sep 17 18:30:21.846944 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.846906    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:21.947113 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:21.947077    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.531513 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:22.531457    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:22.683760 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:22.683728    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1509c79d-9c57-4cb2-80cc-94a55a0d3103" path="/var/lib/kubelet/pods/1509c79d-9c57-4cb2-80cc-94a55a0d3103/volumes"
Sep 17 18:30:23.320870 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:23.320822    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.025270 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:24.025238    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:24.811780 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:24.811747    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:27.517030 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.516978    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:27.631690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.631660    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:27.794026 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.793925    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"]
Sep 17 18:30:27.794280 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.794268    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="628b1a0f-a488-426e-bd20-0a2ced10af4c" containerName="revoke-keys"
Sep 17 18:30:27.794325 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.794282    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="628b1a0f-a488-426e-bd20-0a2ced10af4c" containerName="revoke-keys"
Sep 17 18:30:27.794325 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.794297    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1509c79d-9c57-4cb2-80cc-94a55a0d3103" containerName="payload-processing"
Sep 17 18:30:27.794325 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.794303    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="1509c79d-9c57-4cb2-80cc-94a55a0d3103" containerName="payload-processing"
Sep 17 18:30:27.794424 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.794354    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="628b1a0f-a488-426e-bd20-0a2ced10af4c" containerName="revoke-keys"
Sep 17 18:30:27.794424 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.794365    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="1509c79d-9c57-4cb2-80cc-94a55a0d3103" containerName="payload-processing"
Sep 17 18:30:27.798609 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.798592    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"
Sep 17 18:30:27.801536 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.801515    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:30:27.801681 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.801516    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:30:27.810247 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.810226    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"]
Sep 17 18:30:27.885199 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.885167    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039b3686-331b-4208-bd89-c511daa62efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm\" (UID: \"039b3686-331b-4208-bd89-c511daa62efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"
Sep 17 18:30:27.974888 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.974854    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:27.986164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.986142    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039b3686-331b-4208-bd89-c511daa62efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm\" (UID: \"039b3686-331b-4208-bd89-c511daa62efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"
Sep 17 18:30:27.986777 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:27.986752    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039b3686-331b-4208-bd89-c511daa62efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm\" (UID: \"039b3686-331b-4208-bd89-c511daa62efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"
Sep 17 18:30:28.109154 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:28.109056    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"
Sep 17 18:30:28.381513 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:28.381488    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"]
Sep 17 18:30:28.383730 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:30:28.383704    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod039b3686_331b_4208_bd89_c511daa62efc.slice/crio-d6bf763f32c906657c4c37e28d62283b565d2a611f967d88a6ddb181e729c30a WatchSource:0}: Error finding container d6bf763f32c906657c4c37e28d62283b565d2a611f967d88a6ddb181e729c30a: Status 404 returned error can't find the container with id d6bf763f32c906657c4c37e28d62283b565d2a611f967d88a6ddb181e729c30a
Sep 17 18:30:28.421526 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:28.421502    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:30:29.143533 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:29.143503    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:29.337397 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:29.337315    2589 generic.go:356] "Generic (PLEG): container finished" podID="039b3686-331b-4208-bd89-c511daa62efc" containerID="f454513b6466736453b8fe32300246e110d53c38ebbb32a6198e4d8f6c0840ea" exitCode=0
Sep 17 18:30:29.337538 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:29.337402    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm" event={"ID":"039b3686-331b-4208-bd89-c511daa62efc","Type":"ContainerDied","Data":"f454513b6466736453b8fe32300246e110d53c38ebbb32a6198e4d8f6c0840ea"}
Sep 17 18:30:29.337538 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:29.337458    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm" event={"ID":"039b3686-331b-4208-bd89-c511daa62efc","Type":"ContainerStarted","Data":"d6bf763f32c906657c4c37e28d62283b565d2a611f967d88a6ddb181e729c30a"}
Sep 17 18:30:29.567625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:29.567594    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.341154 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:30.341116    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:30.473348 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:30.473324    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"
Sep 17 18:30:30.508221 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:30.508197    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039b3686-331b-4208-bd89-c511daa62efc-maas-api-service-ca\") pod \"039b3686-331b-4208-bd89-c511daa62efc\" (UID: \"039b3686-331b-4208-bd89-c511daa62efc\") "
Sep 17 18:30:30.508507 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:30.508484    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/039b3686-331b-4208-bd89-c511daa62efc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "039b3686-331b-4208-bd89-c511daa62efc" (UID: "039b3686-331b-4208-bd89-c511daa62efc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:30:30.608926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:30.608843    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/039b3686-331b-4208-bd89-c511daa62efc-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:31.298846 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:31.298807    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:31.348671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:31.348639    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm" event={"ID":"039b3686-331b-4208-bd89-c511daa62efc","Type":"ContainerDied","Data":"d6bf763f32c906657c4c37e28d62283b565d2a611f967d88a6ddb181e729c30a"}
Sep 17 18:30:31.348671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:31.348673    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"
Sep 17 18:30:31.349231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:31.348709    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6bf763f32c906657c4c37e28d62283b565d2a611f967d88a6ddb181e729c30a"
Sep 17 18:30:32.023628 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.023591    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf"]
Sep 17 18:30:32.026427 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.026402    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-5d8053-29827830-l7pgf"]
Sep 17 18:30:32.032112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.032084    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:32.123594 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.123559    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:32.341803 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.341716    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:32.439077 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.439038    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:32.492874 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.492844    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"]
Sep 17 18:30:32.499676 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.499648    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-5d8053-l4qqp"]
Sep 17 18:30:32.652182 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.652078    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:32.683848 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.683815    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="628b1a0f-a488-426e-bd20-0a2ced10af4c" path="/var/lib/kubelet/pods/628b1a0f-a488-426e-bd20-0a2ced10af4c/volumes"
Sep 17 18:30:32.684144 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.684131    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1e2af21-c859-415e-a596-a24f80f46ac0" path="/var/lib/kubelet/pods/a1e2af21-c859-415e-a596-a24f80f46ac0/volumes"
Sep 17 18:30:32.780621 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:32.780594    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:33.344211 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.344166    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g" podUID="533e9753-039d-44b4-bada-b6042f56cac9" containerName="main" containerID="cri-o://1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934" gracePeriod=45
Sep 17 18:30:33.353932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.353898    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb" podUID="694cc9d0-dafe-4833-a7b8-e70e1eb7a662" containerName="main" containerID="cri-o://5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85" gracePeriod=45
Sep 17 18:30:33.359818 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.359785    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9" podUID="6c7ce848-838c-4416-9e96-0c6ad8208f50" containerName="main" containerID="cri-o://22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8" gracePeriod=45
Sep 17 18:30:33.643282 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.643252    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:30:33.647247 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.647222    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:30:33.654272 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.654251    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:30:33.734864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.734836    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tmp-dir\") pod \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") "
Sep 17 18:30:33.735084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.734870    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-tmp-dir\") pod \"533e9753-039d-44b4-bada-b6042f56cac9\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") "
Sep 17 18:30:33.735084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.734898    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-home\") pod \"6c7ce848-838c-4416-9e96-0c6ad8208f50\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") "
Sep 17 18:30:33.735084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.734919    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jrbr\" (UniqueName: \"kubernetes.io/projected/6c7ce848-838c-4416-9e96-0c6ad8208f50-kube-api-access-9jrbr\") pod \"6c7ce848-838c-4416-9e96-0c6ad8208f50\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") "
Sep 17 18:30:33.735084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.734944    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-dshm\") pod \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") "
Sep 17 18:30:33.735084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.734973    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-dshm\") pod \"6c7ce848-838c-4416-9e96-0c6ad8208f50\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") "
Sep 17 18:30:33.735084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.734994    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7ce848-838c-4416-9e96-0c6ad8208f50-tls-certs\") pod \"6c7ce848-838c-4416-9e96-0c6ad8208f50\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") "
Sep 17 18:30:33.735084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735042    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-home\") pod \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") "
Sep 17 18:30:33.735084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735066    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-model-cache\") pod \"533e9753-039d-44b4-bada-b6042f56cac9\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") "
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735114    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tls-certs\") pod \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") "
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735137    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-tmp-dir\") pod \"6c7ce848-838c-4416-9e96-0c6ad8208f50\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") "
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735162    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "694cc9d0-dafe-4833-a7b8-e70e1eb7a662" (UID: "694cc9d0-dafe-4833-a7b8-e70e1eb7a662"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735175    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "533e9753-039d-44b4-bada-b6042f56cac9" (UID: "533e9753-039d-44b4-bada-b6042f56cac9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735177    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-home" (OuterVolumeSpecName: "home") pod "6c7ce848-838c-4416-9e96-0c6ad8208f50" (UID: "6c7ce848-838c-4416-9e96-0c6ad8208f50"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735197    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533e9753-039d-44b4-bada-b6042f56cac9-tls-certs\") pod \"533e9753-039d-44b4-bada-b6042f56cac9\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") "
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735362    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-model-cache" (OuterVolumeSpecName: "model-cache") pod "533e9753-039d-44b4-bada-b6042f56cac9" (UID: "533e9753-039d-44b4-bada-b6042f56cac9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735396    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6c7ce848-838c-4416-9e96-0c6ad8208f50" (UID: "6c7ce848-838c-4416-9e96-0c6ad8208f50"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735440    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-dshm\") pod \"533e9753-039d-44b4-bada-b6042f56cac9\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") "
Sep 17 18:30:33.735481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735480    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-home\") pod \"533e9753-039d-44b4-bada-b6042f56cac9\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") "
Sep 17 18:30:33.736206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735509    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-model-cache\") pod \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") "
Sep 17 18:30:33.736206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735553    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vhfct\" (UniqueName: \"kubernetes.io/projected/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-kube-api-access-vhfct\") pod \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\" (UID: \"694cc9d0-dafe-4833-a7b8-e70e1eb7a662\") "
Sep 17 18:30:33.736206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735578    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-model-cache\") pod \"6c7ce848-838c-4416-9e96-0c6ad8208f50\" (UID: \"6c7ce848-838c-4416-9e96-0c6ad8208f50\") "
Sep 17 18:30:33.736206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735615    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4pfxm\" (UniqueName: \"kubernetes.io/projected/533e9753-039d-44b4-bada-b6042f56cac9-kube-api-access-4pfxm\") pod \"533e9753-039d-44b4-bada-b6042f56cac9\" (UID: \"533e9753-039d-44b4-bada-b6042f56cac9\") "
Sep 17 18:30:33.736206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.735808    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-home" (OuterVolumeSpecName: "home") pod "694cc9d0-dafe-4833-a7b8-e70e1eb7a662" (UID: "694cc9d0-dafe-4833-a7b8-e70e1eb7a662"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.736206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736087    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-model-cache" (OuterVolumeSpecName: "model-cache") pod "694cc9d0-dafe-4833-a7b8-e70e1eb7a662" (UID: "694cc9d0-dafe-4833-a7b8-e70e1eb7a662"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.736509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736418    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.736509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736442    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.736509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736456    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.736509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736470    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.736509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736482    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.736509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736495    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.736509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736508    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.736857 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736738    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-home" (OuterVolumeSpecName: "home") pod "533e9753-039d-44b4-bada-b6042f56cac9" (UID: "533e9753-039d-44b4-bada-b6042f56cac9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.736980 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.736956    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-model-cache" (OuterVolumeSpecName: "model-cache") pod "6c7ce848-838c-4416-9e96-0c6ad8208f50" (UID: "6c7ce848-838c-4416-9e96-0c6ad8208f50"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.738369 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.738335    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-dshm" (OuterVolumeSpecName: "dshm") pod "694cc9d0-dafe-4833-a7b8-e70e1eb7a662" (UID: "694cc9d0-dafe-4833-a7b8-e70e1eb7a662"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.738529 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.738494    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7ce848-838c-4416-9e96-0c6ad8208f50-kube-api-access-9jrbr" (OuterVolumeSpecName: "kube-api-access-9jrbr") pod "6c7ce848-838c-4416-9e96-0c6ad8208f50" (UID: "6c7ce848-838c-4416-9e96-0c6ad8208f50"). InnerVolumeSpecName "kube-api-access-9jrbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:33.738782 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.738745    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-dshm" (OuterVolumeSpecName: "dshm") pod "6c7ce848-838c-4416-9e96-0c6ad8208f50" (UID: "6c7ce848-838c-4416-9e96-0c6ad8208f50"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.738997 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.738975    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533e9753-039d-44b4-bada-b6042f56cac9-kube-api-access-4pfxm" (OuterVolumeSpecName: "kube-api-access-4pfxm") pod "533e9753-039d-44b4-bada-b6042f56cac9" (UID: "533e9753-039d-44b4-bada-b6042f56cac9"). InnerVolumeSpecName "kube-api-access-4pfxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:33.739284 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.739257    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/533e9753-039d-44b4-bada-b6042f56cac9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "533e9753-039d-44b4-bada-b6042f56cac9" (UID: "533e9753-039d-44b4-bada-b6042f56cac9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:33.739393 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.739373    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "694cc9d0-dafe-4833-a7b8-e70e1eb7a662" (UID: "694cc9d0-dafe-4833-a7b8-e70e1eb7a662"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:33.739497 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.739482    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-kube-api-access-vhfct" (OuterVolumeSpecName: "kube-api-access-vhfct") pod "694cc9d0-dafe-4833-a7b8-e70e1eb7a662" (UID: "694cc9d0-dafe-4833-a7b8-e70e1eb7a662"). InnerVolumeSpecName "kube-api-access-vhfct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:30:33.739707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.739693    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c7ce848-838c-4416-9e96-0c6ad8208f50-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6c7ce848-838c-4416-9e96-0c6ad8208f50" (UID: "6c7ce848-838c-4416-9e96-0c6ad8208f50"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:30:33.740179 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.740151    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-dshm" (OuterVolumeSpecName: "dshm") pod "533e9753-039d-44b4-bada-b6042f56cac9" (UID: "533e9753-039d-44b4-bada-b6042f56cac9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:30:33.837168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837139    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837164    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837168 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837172    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c7ce848-838c-4416-9e96-0c6ad8208f50-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837181    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837189    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/533e9753-039d-44b4-bada-b6042f56cac9-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837196    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837202    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/533e9753-039d-44b4-bada-b6042f56cac9-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837211    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhfct\" (UniqueName: \"kubernetes.io/projected/694cc9d0-dafe-4833-a7b8-e70e1eb7a662-kube-api-access-vhfct\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837224    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c7ce848-838c-4416-9e96-0c6ad8208f50-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837235    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pfxm\" (UniqueName: \"kubernetes.io/projected/533e9753-039d-44b4-bada-b6042f56cac9-kube-api-access-4pfxm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.837413 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.837243    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jrbr\" (UniqueName: \"kubernetes.io/projected/6c7ce848-838c-4416-9e96-0c6ad8208f50-kube-api-access-9jrbr\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:30:33.856951 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:33.856918    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:34.366743 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.366707    2589 generic.go:356] "Generic (PLEG): container finished" podID="6c7ce848-838c-4416-9e96-0c6ad8208f50" containerID="22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8" exitCode=0
Sep 17 18:30:34.366875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.366754    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"
Sep 17 18:30:34.366875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.366791    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9" event={"ID":"6c7ce848-838c-4416-9e96-0c6ad8208f50","Type":"ContainerDied","Data":"22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8"}
Sep 17 18:30:34.366875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.366834    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9" event={"ID":"6c7ce848-838c-4416-9e96-0c6ad8208f50","Type":"ContainerDied","Data":"091aa732480af87fb1ec5ee8aceedae76d0459d7be63a3813d7f6b24b1f7b04b"}
Sep 17 18:30:34.366875 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.366857    2589 scope.go:117] "RemoveContainer" containerID="22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8"
Sep 17 18:30:34.368888 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.368855    2589 generic.go:356] "Generic (PLEG): container finished" podID="533e9753-039d-44b4-bada-b6042f56cac9" containerID="1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934" exitCode=0
Sep 17 18:30:34.369027 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.368907    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"
Sep 17 18:30:34.369027 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.368946    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g" event={"ID":"533e9753-039d-44b4-bada-b6042f56cac9","Type":"ContainerDied","Data":"1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934"}
Sep 17 18:30:34.369027 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.368983    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g" event={"ID":"533e9753-039d-44b4-bada-b6042f56cac9","Type":"ContainerDied","Data":"f9a66931205015224f861b67ca384faa6c2c8d7e9723993a2584202d4ecfbf54"}
Sep 17 18:30:34.370683 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.370665    2589 generic.go:356] "Generic (PLEG): container finished" podID="694cc9d0-dafe-4833-a7b8-e70e1eb7a662" containerID="5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85" exitCode=0
Sep 17 18:30:34.370781 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.370698    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"
Sep 17 18:30:34.370849 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.370700    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb" event={"ID":"694cc9d0-dafe-4833-a7b8-e70e1eb7a662","Type":"ContainerDied","Data":"5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85"}
Sep 17 18:30:34.370907 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.370853    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb" event={"ID":"694cc9d0-dafe-4833-a7b8-e70e1eb7a662","Type":"ContainerDied","Data":"e266825221d3dd7c184152fb2c7df8641cb2bc59a998a31574ad8ca4cd0375e7"}
Sep 17 18:30:34.378820 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.378794    2589 scope.go:117] "RemoveContainer" containerID="22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8"
Sep 17 18:30:34.379207 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:30:34.379130    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8\": container with ID starting with 22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8 not found: ID does not exist" containerID="22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8"
Sep 17 18:30:34.379207 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.379166    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8"} err="failed to get container status \"22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8\": rpc error: code = NotFound desc = could not find container \"22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8\": container with ID starting with 22832c4a95397db9a4b11bffd2f69e96706379538d78ec3347b3e160544cecd8 not found: ID does not exist"
Sep 17 18:30:34.379207 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.379190    2589 scope.go:117] "RemoveContainer" containerID="1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934"
Sep 17 18:30:34.387553 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.387539    2589 scope.go:117] "RemoveContainer" containerID="1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934"
Sep 17 18:30:34.387789 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:30:34.387775    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934\": container with ID starting with 1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934 not found: ID does not exist" containerID="1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934"
Sep 17 18:30:34.387828 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.387798    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934"} err="failed to get container status \"1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934\": rpc error: code = NotFound desc = could not find container \"1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934\": container with ID starting with 1e2e8c0c6434b881804f11627dd065355c759cd8878599cf4b3bebc84414a934 not found: ID does not exist"
Sep 17 18:30:34.387828 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.387814    2589 scope.go:117] "RemoveContainer" containerID="5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85"
Sep 17 18:30:34.392785 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.392762    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"]
Sep 17 18:30:34.396605 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.396584    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/embedding-w0-5d8053-kserve-6786f687f9-gqt8g"]
Sep 17 18:30:34.396861 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.396846    2589 scope.go:117] "RemoveContainer" containerID="5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85"
Sep 17 18:30:34.397149 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:30:34.397132    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85\": container with ID starting with 5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85 not found: ID does not exist" containerID="5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85"
Sep 17 18:30:34.397205 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.397156    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85"} err="failed to get container status \"5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85\": rpc error: code = NotFound desc = could not find container \"5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85\": container with ID starting with 5e492da7c9fa6eef7770cd6a0c41a5c153be2f24ef5d683002f086e0f4f3eb85 not found: ID does not exist"
Sep 17 18:30:34.405334 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.405311    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"]
Sep 17 18:30:34.409606 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.409585    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/simulator-w0-5d8053-kserve-8458c65846-jsvm9"]
Sep 17 18:30:34.419692 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.419672    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"]
Sep 17 18:30:34.423790 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.423771    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-5d8053/premium-w0-5d8053-kserve-69bfd4d48c-mm6qb"]
Sep 17 18:30:34.661137 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.661038    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:34.684130 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.684098    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="533e9753-039d-44b4-bada-b6042f56cac9" path="/var/lib/kubelet/pods/533e9753-039d-44b4-bada-b6042f56cac9/volumes"
Sep 17 18:30:34.684535 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.684522    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="694cc9d0-dafe-4833-a7b8-e70e1eb7a662" path="/var/lib/kubelet/pods/694cc9d0-dafe-4833-a7b8-e70e1eb7a662/volumes"
Sep 17 18:30:34.684907 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:34.684895    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c7ce848-838c-4416-9e96-0c6ad8208f50" path="/var/lib/kubelet/pods/6c7ce848-838c-4416-9e96-0c6ad8208f50/volumes"
Sep 17 18:30:35.428986 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:35.428946    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:36.257698 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:36.257659    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:37.038804 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:37.038773    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.100655 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:38.100618    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.759622 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:38.759587    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.834662 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:38.834610    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.933343 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:38.933299    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:38.980552 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:38.980525    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"]
Sep 17 18:30:38.986645 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:38.986617    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-1a9245-9s9jm"]
Sep 17 18:30:39.014482 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:39.014396    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:39.349365 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:39.349283    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:39.449947 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:39.449912    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:39.599721 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:39.599632    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:39.671618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:39.671586    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.065648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:40.065618    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.374752 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:40.374658    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:40.683881 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:40.683803    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="039b3686-331b-4208-bd89-c511daa62efc" path="/var/lib/kubelet/pods/039b3686-331b-4208-bd89-c511daa62efc/volumes"
Sep 17 18:30:41.552484 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:41.552445    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:42.382340 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:42.382303    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:45.787405 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:45.787371    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:45.864230 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:45.864194    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:46.063613 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:46.063528    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:46.709496 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:46.709464    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:47.706090 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:47.706052    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:48.414434 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:48.414391    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:49.236543 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:49.236498    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:49.940781 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:49.940751    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:55.398381 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:55.398337    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:55.457163 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:55.457121    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:56.003452 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:56.003413    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.047887 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.047850    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.048312 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.048210    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx"]
Sep 17 18:30:57.049783 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.049764    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-03270e-29827830-5v6vx"]
Sep 17 18:30:57.119814 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.119785    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.184417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.184386    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.201409 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.201375    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:30:57.270422 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.270390    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.335668 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.335562    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.622962 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.622868    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:57.722166 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:57.722131    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:58.465316 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:58.465288    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:30:58.683656 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:58.683626    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08ce5059-1fbb-4871-aa0a-0133da21c633" path="/var/lib/kubelet/pods/08ce5059-1fbb-4871-aa0a-0133da21c633/volumes"
Sep 17 18:30:59.261940 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:30:59.261902    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:02.066965 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:02.066929    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:03.554521 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:03.554474    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:03.650213 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:03.650167    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:03.787810 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:03.787776    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"]
Sep 17 18:31:03.788048 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:03.787981    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy" containerID="cri-o://0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea" gracePeriod=30
Sep 17 18:31:03.809569 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:03.809479    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:05.101708 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:05.101661    2589 patch_prober.go:28] interesting pod/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 18:31:05.102086 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:05.101741    2589 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:31:08.303295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:08.303260    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:09.028141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.028116    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:31:09.158924 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.158843    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0295ec94-779d-491c-a732-aba39704f88e-istio-podinfo\") pod \"0295ec94-779d-491c-a732-aba39704f88e\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") "
Sep 17 18:31:09.158924 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.158884    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psxq2\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-kube-api-access-psxq2\") pod \"0295ec94-779d-491c-a732-aba39704f88e\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") "
Sep 17 18:31:09.158924 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.158907    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-envoy\") pod \"0295ec94-779d-491c-a732-aba39704f88e\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") "
Sep 17 18:31:09.159255 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.158929    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-credential-socket\") pod \"0295ec94-779d-491c-a732-aba39704f88e\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") "
Sep 17 18:31:09.159255 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.158965    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0295ec94-779d-491c-a732-aba39704f88e-istiod-ca-cert\") pod \"0295ec94-779d-491c-a732-aba39704f88e\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") "
Sep 17 18:31:09.159255 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.158987    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-socket\") pod \"0295ec94-779d-491c-a732-aba39704f88e\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") "
Sep 17 18:31:09.159255 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.159048    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-data\") pod \"0295ec94-779d-491c-a732-aba39704f88e\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") "
Sep 17 18:31:09.159255 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.159115    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-certs\") pod \"0295ec94-779d-491c-a732-aba39704f88e\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") "
Sep 17 18:31:09.159255 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.159144    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-istio-token\") pod \"0295ec94-779d-491c-a732-aba39704f88e\" (UID: \"0295ec94-779d-491c-a732-aba39704f88e\") "
Sep 17 18:31:09.159577 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.159363    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0295ec94-779d-491c-a732-aba39704f88e" (UID: "0295ec94-779d-491c-a732-aba39704f88e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:09.159577 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.159409    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-data" (OuterVolumeSpecName: "istio-data") pod "0295ec94-779d-491c-a732-aba39704f88e" (UID: "0295ec94-779d-491c-a732-aba39704f88e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:09.159577 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.159441    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0295ec94-779d-491c-a732-aba39704f88e" (UID: "0295ec94-779d-491c-a732-aba39704f88e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:09.159577 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.159491    2589 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:09.159961 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.159622    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0295ec94-779d-491c-a732-aba39704f88e" (UID: "0295ec94-779d-491c-a732-aba39704f88e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:09.159961 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.159831    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0295ec94-779d-491c-a732-aba39704f88e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0295ec94-779d-491c-a732-aba39704f88e" (UID: "0295ec94-779d-491c-a732-aba39704f88e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:31:09.161335 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.161312    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0295ec94-779d-491c-a732-aba39704f88e" (UID: "0295ec94-779d-491c-a732-aba39704f88e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:09.161458 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.161348    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0295ec94-779d-491c-a732-aba39704f88e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0295ec94-779d-491c-a732-aba39704f88e" (UID: "0295ec94-779d-491c-a732-aba39704f88e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:31:09.161458 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.161359    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-kube-api-access-psxq2" (OuterVolumeSpecName: "kube-api-access-psxq2") pod "0295ec94-779d-491c-a732-aba39704f88e" (UID: "0295ec94-779d-491c-a732-aba39704f88e"). InnerVolumeSpecName "kube-api-access-psxq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:09.161561 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.161548    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-istio-token" (OuterVolumeSpecName: "istio-token") pod "0295ec94-779d-491c-a732-aba39704f88e" (UID: "0295ec94-779d-491c-a732-aba39704f88e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:09.260237 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.260205    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psxq2\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-kube-api-access-psxq2\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:09.260237 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.260234    2589 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:09.260237 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.260243    2589 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:09.260470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.260252    2589 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0295ec94-779d-491c-a732-aba39704f88e-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:09.260470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.260260    2589 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:09.260470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.260268    2589 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0295ec94-779d-491c-a732-aba39704f88e-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:09.260470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.260276    2589 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0295ec94-779d-491c-a732-aba39704f88e-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:09.260470 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.260283    2589 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0295ec94-779d-491c-a732-aba39704f88e-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:09.562557 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.562526    2589 generic.go:356] "Generic (PLEG): container finished" podID="0295ec94-779d-491c-a732-aba39704f88e" containerID="0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea" exitCode=0
Sep 17 18:31:09.563065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.562578    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"
Sep 17 18:31:09.563065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.562610    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" event={"ID":"0295ec94-779d-491c-a732-aba39704f88e","Type":"ContainerDied","Data":"0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea"}
Sep 17 18:31:09.563065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.562679    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4" event={"ID":"0295ec94-779d-491c-a732-aba39704f88e","Type":"ContainerDied","Data":"0aabbf420c9ee71ea85e51cac0ef944599803d706e7830aa65e8537e62da619e"}
Sep 17 18:31:09.563065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.562714    2589 scope.go:117] "RemoveContainer" containerID="0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea"
Sep 17 18:31:09.572353 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.572338    2589 scope.go:117] "RemoveContainer" containerID="cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42"
Sep 17 18:31:09.580299 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.580283    2589 scope.go:117] "RemoveContainer" containerID="0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea"
Sep 17 18:31:09.580564 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:31:09.580544    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea\": container with ID starting with 0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea not found: ID does not exist" containerID="0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea"
Sep 17 18:31:09.580639 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.580574    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea"} err="failed to get container status \"0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea\": rpc error: code = NotFound desc = could not find container \"0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea\": container with ID starting with 0d9f081fc76e7590e8b985ba8ca331c14c1c191a8fa0b99e5b1be5cfe68c1dea not found: ID does not exist"
Sep 17 18:31:09.580639 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.580598    2589 scope.go:117] "RemoveContainer" containerID="cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42"
Sep 17 18:31:09.580833 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:31:09.580818    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42\": container with ID starting with cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42 not found: ID does not exist" containerID="cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42"
Sep 17 18:31:09.580888 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.580838    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42"} err="failed to get container status \"cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42\": rpc error: code = NotFound desc = could not find container \"cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42\": container with ID starting with cecb749b9be795289551f076fe1562ea73984c71cccb376d6ecea4a9dc106c42 not found: ID does not exist"
Sep 17 18:31:09.593613 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.593589    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"]
Sep 17 18:31:09.606290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:09.606270    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-03270e-openshift-default-6bd7454cc4-rqwq4"]
Sep 17 18:31:10.683547 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:10.683516    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0295ec94-779d-491c-a732-aba39704f88e" path="/var/lib/kubelet/pods/0295ec94-779d-491c-a732-aba39704f88e/volumes"
Sep 17 18:31:11.561827 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:11.561794    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:31:25.692617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:25.692581    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:31:32.132967 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:32.132928    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:33.099301 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:33.099269    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"]
Sep 17 18:31:33.124874 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:33.124843    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:36.700857 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:36.700828    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:31:41.595407 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:41.595369    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:41.655957 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:41.655901    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:43.012290 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:43.012243    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:31:47.416726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:47.416689    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:31:48.118818 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.118769    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj" podUID="66967875-8d1e-4036-9923-baf63df706d5" containerName="main" containerID="cri-o://060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961" gracePeriod=45
Sep 17 18:31:48.371431 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.371370    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:31:48.476236 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476204    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-model-cache\") pod \"66967875-8d1e-4036-9923-baf63df706d5\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") "
Sep 17 18:31:48.476707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476266    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-dshm\") pod \"66967875-8d1e-4036-9923-baf63df706d5\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") "
Sep 17 18:31:48.476707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476326    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66967875-8d1e-4036-9923-baf63df706d5-tls-certs\") pod \"66967875-8d1e-4036-9923-baf63df706d5\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") "
Sep 17 18:31:48.476707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476344    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-home\") pod \"66967875-8d1e-4036-9923-baf63df706d5\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") "
Sep 17 18:31:48.476707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476369    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-tmp-dir\") pod \"66967875-8d1e-4036-9923-baf63df706d5\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") "
Sep 17 18:31:48.476707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476414    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xkq89\" (UniqueName: \"kubernetes.io/projected/66967875-8d1e-4036-9923-baf63df706d5-kube-api-access-xkq89\") pod \"66967875-8d1e-4036-9923-baf63df706d5\" (UID: \"66967875-8d1e-4036-9923-baf63df706d5\") "
Sep 17 18:31:48.476707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476481    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-model-cache" (OuterVolumeSpecName: "model-cache") pod "66967875-8d1e-4036-9923-baf63df706d5" (UID: "66967875-8d1e-4036-9923-baf63df706d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:48.476707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476662    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-home" (OuterVolumeSpecName: "home") pod "66967875-8d1e-4036-9923-baf63df706d5" (UID: "66967875-8d1e-4036-9923-baf63df706d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:48.476707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476672    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "66967875-8d1e-4036-9923-baf63df706d5" (UID: "66967875-8d1e-4036-9923-baf63df706d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:48.476707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.476690    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:48.478718 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.478690    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66967875-8d1e-4036-9923-baf63df706d5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "66967875-8d1e-4036-9923-baf63df706d5" (UID: "66967875-8d1e-4036-9923-baf63df706d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:31:48.478829 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.478738    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-dshm" (OuterVolumeSpecName: "dshm") pod "66967875-8d1e-4036-9923-baf63df706d5" (UID: "66967875-8d1e-4036-9923-baf63df706d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:31:48.478829 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.478752    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66967875-8d1e-4036-9923-baf63df706d5-kube-api-access-xkq89" (OuterVolumeSpecName: "kube-api-access-xkq89") pod "66967875-8d1e-4036-9923-baf63df706d5" (UID: "66967875-8d1e-4036-9923-baf63df706d5"). InnerVolumeSpecName "kube-api-access-xkq89". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:31:48.577759 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.577723    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:48.577759 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.577750    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66967875-8d1e-4036-9923-baf63df706d5-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:48.577759 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.577762    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:48.577759 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.577770    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66967875-8d1e-4036-9923-baf63df706d5-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:48.578083 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.577778    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkq89\" (UniqueName: \"kubernetes.io/projected/66967875-8d1e-4036-9923-baf63df706d5-kube-api-access-xkq89\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:31:48.767471 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.767431    2589 generic.go:356] "Generic (PLEG): container finished" podID="66967875-8d1e-4036-9923-baf63df706d5" containerID="060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961" exitCode=0
Sep 17 18:31:48.767471 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.767476    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"
Sep 17 18:31:48.767702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.767479    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj" event={"ID":"66967875-8d1e-4036-9923-baf63df706d5","Type":"ContainerDied","Data":"060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961"}
Sep 17 18:31:48.767702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.767518    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj" event={"ID":"66967875-8d1e-4036-9923-baf63df706d5","Type":"ContainerDied","Data":"ee976d95d4b1abdbe4f6c35142897fb0674490ec8fe2460267c80c0e51ae7322"}
Sep 17 18:31:48.767702 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.767540    2589 scope.go:117] "RemoveContainer" containerID="060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961"
Sep 17 18:31:48.776248 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.776230    2589 scope.go:117] "RemoveContainer" containerID="060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961"
Sep 17 18:31:48.776518 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:31:48.776499    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961\": container with ID starting with 060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961 not found: ID does not exist" containerID="060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961"
Sep 17 18:31:48.776570 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.776528    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961"} err="failed to get container status \"060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961\": rpc error: code = NotFound desc = could not find container \"060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961\": container with ID starting with 060019e16ff0c00dc6da8e4a641d074dcb0f1e1b3e5bc0dd39c5ab4143f00961 not found: ID does not exist"
Sep 17 18:31:48.782621 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.782594    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"]
Sep 17 18:31:48.788162 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:48.788139    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/auth-test-model-d81d7c-kserve-757b7cd7c5-qvttj"]
Sep 17 18:31:50.685629 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:31:50.685593    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66967875-8d1e-4036-9923-baf63df706d5" path="/var/lib/kubelet/pods/66967875-8d1e-4036-9923-baf63df706d5/volumes"
Sep 17 18:32:04.680733 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:04.680696    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:04.737318 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:04.737286    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:22.102816 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.102780    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"]
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103126    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103138    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103151    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c7ce848-838c-4416-9e96-0c6ad8208f50" containerName="main"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103156    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c7ce848-838c-4416-9e96-0c6ad8208f50" containerName="main"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103165    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="039b3686-331b-4208-bd89-c511daa62efc" containerName="revoke-keys"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103174    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="039b3686-331b-4208-bd89-c511daa62efc" containerName="revoke-keys"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103182    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103187    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103199    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="694cc9d0-dafe-4833-a7b8-e70e1eb7a662" containerName="main"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103204    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="694cc9d0-dafe-4833-a7b8-e70e1eb7a662" containerName="main"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103210    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66967875-8d1e-4036-9923-baf63df706d5" containerName="main"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103214    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="66967875-8d1e-4036-9923-baf63df706d5" containerName="main"
Sep 17 18:32:22.103215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103223    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="533e9753-039d-44b4-bada-b6042f56cac9" containerName="main"
Sep 17 18:32:22.103617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103228    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="533e9753-039d-44b4-bada-b6042f56cac9" containerName="main"
Sep 17 18:32:22.103617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103282    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="694cc9d0-dafe-4833-a7b8-e70e1eb7a662" containerName="main"
Sep 17 18:32:22.103617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103290    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy"
Sep 17 18:32:22.103617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103296    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="533e9753-039d-44b4-bada-b6042f56cac9" containerName="main"
Sep 17 18:32:22.103617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103303    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="039b3686-331b-4208-bd89-c511daa62efc" containerName="revoke-keys"
Sep 17 18:32:22.103617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103310    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="66967875-8d1e-4036-9923-baf63df706d5" containerName="main"
Sep 17 18:32:22.103617 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.103315    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c7ce848-838c-4416-9e96-0c6ad8208f50" containerName="main"
Sep 17 18:32:22.106128 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.106112    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.108524 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.108502    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"openshift-service-ca.crt\""
Sep 17 18:32:22.109482 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.109463    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"sub-test-model-d81d7c-kserve-self-signed-certs\""
Sep 17 18:32:22.109580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.109485    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"default-dockercfg-8sbhg\""
Sep 17 18:32:22.109580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.109466    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"kube-root-ca.crt\""
Sep 17 18:32:22.120304 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.120278    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"]
Sep 17 18:32:22.149215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.149182    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tls-certs\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.149386 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.149256    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-model-cache\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.149386 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.149325    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfnhp\" (UniqueName: \"kubernetes.io/projected/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-kube-api-access-zfnhp\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.149386 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.149364    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tmp-dir\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.149559 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.149391    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-home\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.149559 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.149417    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-dshm\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.149559 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.149426    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:22.250166 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.250136    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfnhp\" (UniqueName: \"kubernetes.io/projected/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-kube-api-access-zfnhp\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.250354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.250198    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tmp-dir\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.250354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.250238    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-home\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.250354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.250261    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-dshm\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.250354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.250307    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tls-certs\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.250580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.250375    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-model-cache\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.250679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.250649    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tmp-dir\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.250810 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.250700    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-home\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.250810 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.250751    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-model-cache\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.252627 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.252607    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-dshm\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.252915 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.252896    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tls-certs\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.259120 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.259100    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfnhp\" (UniqueName: \"kubernetes.io/projected/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-kube-api-access-zfnhp\") pod \"sub-test-model-d81d7c-kserve-588856fc69-nmhnp\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.415648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.415556    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:22.691607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.691579    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"]
Sep 17 18:32:22.691982 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:32:22.691960    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4ba3607f_4e02_4f6b_84dd_ab2a6dd8cb19.slice/crio-5e5868a054c8c34b4c273201bd0b619b56e7c1d5658c0e1a4bcb3ea4167373f4 WatchSource:0}: Error finding container 5e5868a054c8c34b4c273201bd0b619b56e7c1d5658c0e1a4bcb3ea4167373f4: Status 404 returned error can't find the container with id 5e5868a054c8c34b4c273201bd0b619b56e7c1d5658c0e1a4bcb3ea4167373f4
Sep 17 18:32:22.939066 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.938965    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp" event={"ID":"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19","Type":"ContainerStarted","Data":"ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128"}
Sep 17 18:32:22.939066 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.939022    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp" event={"ID":"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19","Type":"ContainerStarted","Data":"5e5868a054c8c34b4c273201bd0b619b56e7c1d5658c0e1a4bcb3ea4167373f4"}
Sep 17 18:32:22.956869 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:22.956819    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp" podStartSLOduration=0.956803938 podStartE2EDuration="956.803938ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:22.955801445 +0000 UTC m=+1778.635487312" watchObservedRunningTime="2026-09-17 18:32:22.956803938 +0000 UTC m=+1778.636489791"
Sep 17 18:32:25.898148 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:25.898059    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:32:32.415735 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:32.415702    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:32.416234 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:32.415759    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:32.428524 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:32.428502    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:32.999433 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:32.999410    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:32:39.608460 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:39.608411    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:32:41.499489 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:41.499452    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:32:46.694522 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:46.694483    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:32:46.694928 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:46.694808    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:32:46.944878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:46.944791    2589 scope.go:117] "RemoveContainer" containerID="fe54ef2e17ffaa25bd35681c6d32faad355a846aed990644baf2919fe74c4010"
Sep 17 18:32:54.505885 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:54.505841    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:32:58.387743 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.387711    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7631-bfb1c5"]
Sep 17 18:32:58.388173 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.388111    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="0295ec94-779d-491c-a732-aba39704f88e" containerName="istio-proxy"
Sep 17 18:32:58.391799 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.391783    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-bfb1c5"
Sep 17 18:32:58.393981 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.393958    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-s5zgf\""
Sep 17 18:32:58.396936 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.396913    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7631-bfb1c5"]
Sep 17 18:32:58.571031 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.570973    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjfjh\" (UniqueName: \"kubernetes.io/projected/f41f16bc-214c-4b3b-938c-774a82a959a9-kube-api-access-wjfjh\") pod \"test-curl-7631-bfb1c5\" (UID: \"f41f16bc-214c-4b3b-938c-774a82a959a9\") " pod="opendatahub/test-curl-7631-bfb1c5"
Sep 17 18:32:58.672122 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.672038    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfjh\" (UniqueName: \"kubernetes.io/projected/f41f16bc-214c-4b3b-938c-774a82a959a9-kube-api-access-wjfjh\") pod \"test-curl-7631-bfb1c5\" (UID: \"f41f16bc-214c-4b3b-938c-774a82a959a9\") " pod="opendatahub/test-curl-7631-bfb1c5"
Sep 17 18:32:58.680292 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.680264    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfjh\" (UniqueName: \"kubernetes.io/projected/f41f16bc-214c-4b3b-938c-774a82a959a9-kube-api-access-wjfjh\") pod \"test-curl-7631-bfb1c5\" (UID: \"f41f16bc-214c-4b3b-938c-774a82a959a9\") " pod="opendatahub/test-curl-7631-bfb1c5"
Sep 17 18:32:58.701740 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.701708    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-bfb1c5"
Sep 17 18:32:58.971978 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:58.971952    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7631-bfb1c5"]
Sep 17 18:32:58.974271 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:32:58.974236    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf41f16bc_214c_4b3b_938c_774a82a959a9.slice/crio-ce81b4539f380ec0b264b10467c737368d0f1377b83057548e9f0c3e79b88fa4 WatchSource:0}: Error finding container ce81b4539f380ec0b264b10467c737368d0f1377b83057548e9f0c3e79b88fa4: Status 404 returned error can't find the container with id ce81b4539f380ec0b264b10467c737368d0f1377b83057548e9f0c3e79b88fa4
Sep 17 18:32:59.120417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:59.120367    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7631-bfb1c5" event={"ID":"f41f16bc-214c-4b3b-938c-774a82a959a9","Type":"ContainerStarted","Data":"fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567"}
Sep 17 18:32:59.120417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:59.120413    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7631-bfb1c5" event={"ID":"f41f16bc-214c-4b3b-938c-774a82a959a9","Type":"ContainerStarted","Data":"ce81b4539f380ec0b264b10467c737368d0f1377b83057548e9f0c3e79b88fa4"}
Sep 17 18:32:59.136990 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:59.136941    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7631-bfb1c5" podStartSLOduration=1.136926949 podStartE2EDuration="1.136926949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:32:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:32:59.135658296 +0000 UTC m=+1814.815344152" watchObservedRunningTime="2026-09-17 18:32:59.136926949 +0000 UTC m=+1814.816612802"
Sep 17 18:32:59.714430 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:59.714342    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7631-bfb1c5"]
Sep 17 18:32:59.718445 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:32:59.718408    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7631-bfb1c5"]
Sep 17 18:33:00.124575 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:33:00.124546    2589 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7631-bfb1c5" container="test-curl-7631-bfb1c5"
Sep 17 18:33:01.129218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:01.129177    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7631-bfb1c5" podUID="f41f16bc-214c-4b3b-938c-774a82a959a9" containerName="test-curl-7631-bfb1c5" containerID="cri-o://fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567" gracePeriod=2
Sep 17 18:33:01.732796 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:01.732760    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:02.690926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:02.690894    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"]
Sep 17 18:33:02.712509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:02.712470    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:03.297018 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:03.296973    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-bfb1c5"
Sep 17 18:33:03.417023 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:03.416922    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wjfjh\" (UniqueName: \"kubernetes.io/projected/f41f16bc-214c-4b3b-938c-774a82a959a9-kube-api-access-wjfjh\") pod \"f41f16bc-214c-4b3b-938c-774a82a959a9\" (UID: \"f41f16bc-214c-4b3b-938c-774a82a959a9\") "
Sep 17 18:33:03.419161 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:03.419137    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f41f16bc-214c-4b3b-938c-774a82a959a9-kube-api-access-wjfjh" (OuterVolumeSpecName: "kube-api-access-wjfjh") pod "f41f16bc-214c-4b3b-938c-774a82a959a9" (UID: "f41f16bc-214c-4b3b-938c-774a82a959a9"). InnerVolumeSpecName "kube-api-access-wjfjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:03.517763 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:03.517726    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjfjh\" (UniqueName: \"kubernetes.io/projected/f41f16bc-214c-4b3b-938c-774a82a959a9-kube-api-access-wjfjh\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:33:04.145066 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:04.145032    2589 generic.go:356] "Generic (PLEG): container finished" podID="f41f16bc-214c-4b3b-938c-774a82a959a9" containerID="fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567" exitCode=137
Sep 17 18:33:04.145066 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:04.145064    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7631-bfb1c5"
Sep 17 18:33:04.145614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:04.145120    2589 scope.go:117] "RemoveContainer" containerID="fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567"
Sep 17 18:33:04.154379 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:04.154360    2589 scope.go:117] "RemoveContainer" containerID="fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567"
Sep 17 18:33:04.154693 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:33:04.154672    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567\": container with ID starting with fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567 not found: ID does not exist" containerID="fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567"
Sep 17 18:33:04.154753 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:04.154703    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567"} err="failed to get container status \"fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567\": rpc error: code = NotFound desc = could not find container \"fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567\": container with ID starting with fefac4d1aaa1172f54c61dd9d76c97166419c9c516a6ea48576da0483fc56567 not found: ID does not exist"
Sep 17 18:33:04.155481 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:33:04.155459    2589 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7631-bfb1c5\" is forbidden: User \"system:node:ip-10-0-142-105.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-142-105.ec2.internal' and this object" podUID="f41f16bc-214c-4b3b-938c-774a82a959a9" pod="opendatahub/test-curl-7631-bfb1c5"
Sep 17 18:33:04.683618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:04.683585    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f41f16bc-214c-4b3b-938c-774a82a959a9" path="/var/lib/kubelet/pods/f41f16bc-214c-4b3b-938c-774a82a959a9/volumes"
Sep 17 18:33:05.305814 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:05.305768    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:33:09.324215 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:09.324176    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:17.709915 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:17.709858    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp" podUID="4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" containerName="main" containerID="cri-o://ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128" gracePeriod=45
Sep 17 18:33:17.963706 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:17.963648    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:33:18.037855 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.037821    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-model-cache\") pod \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") "
Sep 17 18:33:18.038069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.037877    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tls-certs\") pod \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") "
Sep 17 18:33:18.038069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.037911    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-dshm\") pod \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") "
Sep 17 18:33:18.038069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.037941    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfnhp\" (UniqueName: \"kubernetes.io/projected/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-kube-api-access-zfnhp\") pod \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") "
Sep 17 18:33:18.038069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.037968    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tmp-dir\") pod \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") "
Sep 17 18:33:18.038069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.038035    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-home\") pod \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\" (UID: \"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19\") "
Sep 17 18:33:18.038367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.038162    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-model-cache" (OuterVolumeSpecName: "model-cache") pod "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" (UID: "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:18.038367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.038289    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:33:18.038367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.038303    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" (UID: "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:18.038367 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.038322    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-home" (OuterVolumeSpecName: "home") pod "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" (UID: "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:18.040370 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.040345    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-dshm" (OuterVolumeSpecName: "dshm") pod "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" (UID: "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:33:18.040460 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.040386    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-kube-api-access-zfnhp" (OuterVolumeSpecName: "kube-api-access-zfnhp") pod "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" (UID: "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19"). InnerVolumeSpecName "kube-api-access-zfnhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:33:18.040460 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.040440    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" (UID: "4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:33:18.139353 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.139319    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:33:18.139353 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.139351    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfnhp\" (UniqueName: \"kubernetes.io/projected/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-kube-api-access-zfnhp\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:33:18.139569 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.139366    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:33:18.139569 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.139378    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:33:18.139569 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.139388    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:33:18.215914 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.215828    2589 generic.go:356] "Generic (PLEG): container finished" podID="4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" containerID="ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128" exitCode=0
Sep 17 18:33:18.215914 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.215875    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"
Sep 17 18:33:18.215914 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.215884    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp" event={"ID":"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19","Type":"ContainerDied","Data":"ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128"}
Sep 17 18:33:18.216248 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.215916    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp" event={"ID":"4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19","Type":"ContainerDied","Data":"5e5868a054c8c34b4c273201bd0b619b56e7c1d5658c0e1a4bcb3ea4167373f4"}
Sep 17 18:33:18.216248 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.215933    2589 scope.go:117] "RemoveContainer" containerID="ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128"
Sep 17 18:33:18.228632 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.228610    2589 scope.go:117] "RemoveContainer" containerID="ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128"
Sep 17 18:33:18.228915 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:33:18.228895    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128\": container with ID starting with ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128 not found: ID does not exist" containerID="ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128"
Sep 17 18:33:18.228974 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.228927    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128"} err="failed to get container status \"ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128\": rpc error: code = NotFound desc = could not find container \"ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128\": container with ID starting with ed5b6f63a2de67607511eddb745d5cea624e74cb0c9f86eecace114da6760128 not found: ID does not exist"
Sep 17 18:33:18.242225 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.242200    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"]
Sep 17 18:33:18.246513 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.246493    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/sub-test-model-d81d7c-kserve-588856fc69-nmhnp"]
Sep 17 18:33:18.683807 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:18.683771    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" path="/var/lib/kubelet/pods/4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19/volumes"
Sep 17 18:33:22.260596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:22.260563    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:22.325553 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:22.325517    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:28.839102 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:28.839070    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:31.827942 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:31.827905    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:33:58.414263 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.414181    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"]
Sep 17 18:33:58.414700 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.414681    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f41f16bc-214c-4b3b-938c-774a82a959a9" containerName="test-curl-7631-bfb1c5"
Sep 17 18:33:58.414749 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.414705    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="f41f16bc-214c-4b3b-938c-774a82a959a9" containerName="test-curl-7631-bfb1c5"
Sep 17 18:33:58.414749 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.414736    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" containerName="main"
Sep 17 18:33:58.414749 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.414744    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" containerName="main"
Sep 17 18:33:58.414869 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.414841    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="f41f16bc-214c-4b3b-938c-774a82a959a9" containerName="test-curl-7631-bfb1c5"
Sep 17 18:33:58.414908 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.414885    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ba3607f-4e02-4f6b-84dd-ab2a6dd8cb19" containerName="main"
Sep 17 18:33:58.419509 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.419491    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.423074 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.423041    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"rate-test-model-d81d7c-kserve-self-signed-certs\""
Sep 17 18:33:58.423196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.423051    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"default-dockercfg-8sbhg\""
Sep 17 18:33:58.423196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.423052    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"kube-root-ca.crt\""
Sep 17 18:33:58.423196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.423108    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-d81d7c\"/\"openshift-service-ca.crt\""
Sep 17 18:33:58.425995 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.425975    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"]
Sep 17 18:33:58.448529 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.448501    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:33:58.501103 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.501067    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-tmp-dir\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.501275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.501112    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-dshm\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.501275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.501128    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-model-cache\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.501275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.501182    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-home\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.501275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.501217    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vsrd\" (UniqueName: \"kubernetes.io/projected/03809a9a-f12f-4df5-8b6d-73f4561c6906-kube-api-access-9vsrd\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.501443 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.501277    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03809a9a-f12f-4df5-8b6d-73f4561c6906-tls-certs\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.601884 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.601847    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-dshm\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.601884 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.601881    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-model-cache\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.602164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.601900    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-home\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.602164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.602033    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vsrd\" (UniqueName: \"kubernetes.io/projected/03809a9a-f12f-4df5-8b6d-73f4561c6906-kube-api-access-9vsrd\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.602164 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.602123    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03809a9a-f12f-4df5-8b6d-73f4561c6906-tls-certs\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.602324 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.602206    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-tmp-dir\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.602324 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.602260    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-model-cache\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.602426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.602404    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-home\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.602524 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.602507    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-tmp-dir\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.604332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.604310    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-dshm\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.604782 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.604762    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03809a9a-f12f-4df5-8b6d-73f4561c6906-tls-certs\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.611304 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.611283    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vsrd\" (UniqueName: \"kubernetes.io/projected/03809a9a-f12f-4df5-8b6d-73f4561c6906-kube-api-access-9vsrd\") pod \"rate-test-model-d81d7c-kserve-777444ffc9-w4kx6\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") " pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:58.729529 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:58.729450    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:33:59.004510 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:59.004483    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"]
Sep 17 18:33:59.006404 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:33:59.006375    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03809a9a_f12f_4df5_8b6d_73f4561c6906.slice/crio-3c8cf89c867b0e61714707dc17cdacb470aba6d0b454d3ad5f244e40c8947d7e WatchSource:0}: Error finding container 3c8cf89c867b0e61714707dc17cdacb470aba6d0b454d3ad5f244e40c8947d7e: Status 404 returned error can't find the container with id 3c8cf89c867b0e61714707dc17cdacb470aba6d0b454d3ad5f244e40c8947d7e
Sep 17 18:33:59.412719 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:59.412633    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6" event={"ID":"03809a9a-f12f-4df5-8b6d-73f4561c6906","Type":"ContainerStarted","Data":"3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc"}
Sep 17 18:33:59.412719 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:59.412677    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6" event={"ID":"03809a9a-f12f-4df5-8b6d-73f4561c6906","Type":"ContainerStarted","Data":"3c8cf89c867b0e61714707dc17cdacb470aba6d0b454d3ad5f244e40c8947d7e"}
Sep 17 18:33:59.433533 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:33:59.433479    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6" podStartSLOduration=1.433462748 podStartE2EDuration="1.433462748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:33:59.431608965 +0000 UTC m=+1875.111294819" watchObservedRunningTime="2026-09-17 18:33:59.433462748 +0000 UTC m=+1875.113148602"
Sep 17 18:34:02.295811 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:02.295772    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:34:08.730548 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:08.730510    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:34:08.730967 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:08.730559    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:34:08.743093 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:08.743068    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:34:09.469933 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:09.469890    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:34:21.411242 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:21.411207    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:34:32.697347 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:32.697309    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:34:39.599109 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:39.599062    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:40.697116 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:40.697082    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"]
Sep 17 18:34:41.023112 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:41.023080    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:41.162655 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:41.162623    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:41.484378 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:41.484284    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:41.765598 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:41.765566    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:44.866418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:44.866384    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:45.148505 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:45.148405    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:45.232106 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:45.232068    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:34:45.259821 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:45.259784    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:45.797495 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:45.797426    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:46.807746 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:46.807712    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:46.908578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:46.908540    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:47.086170 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.086057    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:47.456287 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.456162    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:47.550368 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.550337    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:47.587846 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.587815    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"]
Sep 17 18:34:47.591060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.591044    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:34:47.593481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.593436    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f72155\""
Sep 17 18:34:47.593594 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.593520    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f72155-dockercfg-fwk5f\""
Sep 17 18:34:47.601197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.601177    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"]
Sep 17 18:34:47.638296 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.638269    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf\" (UID: \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:34:47.638418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.638316    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54d9x\" (UniqueName: \"kubernetes.io/projected/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-kube-api-access-54d9x\") pod \"payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf\" (UID: \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:34:47.739779 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.739703    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf\" (UID: \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:34:47.739779 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.739748    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54d9x\" (UniqueName: \"kubernetes.io/projected/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-kube-api-access-54d9x\") pod \"payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf\" (UID: \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:34:47.740406 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.740386    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf\" (UID: \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:34:47.749334 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.749309    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54d9x\" (UniqueName: \"kubernetes.io/projected/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-kube-api-access-54d9x\") pod \"payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf\" (UID: \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:34:47.901693 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.901639    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:34:47.933647 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:47.933614    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:48.177229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:48.177203    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"]
Sep 17 18:34:48.179309 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:34:48.179280    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3418fd0_a73c_46b3_b119_dc7fdbc34d75.slice/crio-f2384a20d176c15f2daf46912c2b67cc170ca32b456c1fe6e04d345159ae1cc8 WatchSource:0}: Error finding container f2384a20d176c15f2daf46912c2b67cc170ca32b456c1fe6e04d345159ae1cc8: Status 404 returned error can't find the container with id f2384a20d176c15f2daf46912c2b67cc170ca32b456c1fe6e04d345159ae1cc8
Sep 17 18:34:48.641245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:48.641203    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:48.646755 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:48.646721    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf" event={"ID":"d3418fd0-a73c-46b3-b119-dc7fdbc34d75","Type":"ContainerStarted","Data":"53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268"}
Sep 17 18:34:48.646889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:48.646788    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf" event={"ID":"d3418fd0-a73c-46b3-b119-dc7fdbc34d75","Type":"ContainerStarted","Data":"f2384a20d176c15f2daf46912c2b67cc170ca32b456c1fe6e04d345159ae1cc8"}
Sep 17 18:34:48.646889 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:48.646818    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:34:48.669422 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:48.669376    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf" podStartSLOduration=1.669362676 podStartE2EDuration="1.669362676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:34:48.667514302 +0000 UTC m=+1924.347200155" watchObservedRunningTime="2026-09-17 18:34:48.669362676 +0000 UTC m=+1924.349048529"
Sep 17 18:34:55.715977 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:55.715915    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6" podUID="03809a9a-f12f-4df5-8b6d-73f4561c6906" containerName="main" containerID="cri-o://3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc" gracePeriod=45
Sep 17 18:34:55.958886 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:55.958861    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:34:56.013481 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.013455    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vsrd\" (UniqueName: \"kubernetes.io/projected/03809a9a-f12f-4df5-8b6d-73f4561c6906-kube-api-access-9vsrd\") pod \"03809a9a-f12f-4df5-8b6d-73f4561c6906\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") "
Sep 17 18:34:56.013663 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.013544    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-dshm\") pod \"03809a9a-f12f-4df5-8b6d-73f4561c6906\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") "
Sep 17 18:34:56.013663 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.013569    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-tmp-dir\") pod \"03809a9a-f12f-4df5-8b6d-73f4561c6906\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") "
Sep 17 18:34:56.013663 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.013614    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-model-cache\") pod \"03809a9a-f12f-4df5-8b6d-73f4561c6906\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") "
Sep 17 18:34:56.013663 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.013649    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03809a9a-f12f-4df5-8b6d-73f4561c6906-tls-certs\") pod \"03809a9a-f12f-4df5-8b6d-73f4561c6906\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") "
Sep 17 18:34:56.013869 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.013678    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-home\") pod \"03809a9a-f12f-4df5-8b6d-73f4561c6906\" (UID: \"03809a9a-f12f-4df5-8b6d-73f4561c6906\") "
Sep 17 18:34:56.013923 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.013879    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03809a9a-f12f-4df5-8b6d-73f4561c6906" (UID: "03809a9a-f12f-4df5-8b6d-73f4561c6906"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:56.013982 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.013887    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-model-cache" (OuterVolumeSpecName: "model-cache") pod "03809a9a-f12f-4df5-8b6d-73f4561c6906" (UID: "03809a9a-f12f-4df5-8b6d-73f4561c6906"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:56.013982 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.013967    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-home" (OuterVolumeSpecName: "home") pod "03809a9a-f12f-4df5-8b6d-73f4561c6906" (UID: "03809a9a-f12f-4df5-8b6d-73f4561c6906"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:56.014120 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.014057    2589 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-tmp-dir\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:34:56.014120 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.014077    2589 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-model-cache\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:34:56.015897 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.015872    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03809a9a-f12f-4df5-8b6d-73f4561c6906-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03809a9a-f12f-4df5-8b6d-73f4561c6906" (UID: "03809a9a-f12f-4df5-8b6d-73f4561c6906"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 17 18:34:56.016291 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.016272    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-dshm" (OuterVolumeSpecName: "dshm") pod "03809a9a-f12f-4df5-8b6d-73f4561c6906" (UID: "03809a9a-f12f-4df5-8b6d-73f4561c6906"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:34:56.016358 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.016284    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03809a9a-f12f-4df5-8b6d-73f4561c6906-kube-api-access-9vsrd" (OuterVolumeSpecName: "kube-api-access-9vsrd") pod "03809a9a-f12f-4df5-8b6d-73f4561c6906" (UID: "03809a9a-f12f-4df5-8b6d-73f4561c6906"). InnerVolumeSpecName "kube-api-access-9vsrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:34:56.115192 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.115158    2589 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03809a9a-f12f-4df5-8b6d-73f4561c6906-tls-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:34:56.115192 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.115184    2589 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-home\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:34:56.115192 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.115194    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vsrd\" (UniqueName: \"kubernetes.io/projected/03809a9a-f12f-4df5-8b6d-73f4561c6906-kube-api-access-9vsrd\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:34:56.115418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.115205    2589 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03809a9a-f12f-4df5-8b6d-73f4561c6906-dshm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:34:56.300093 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.299990    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:34:56.691892 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.691806    2589 generic.go:356] "Generic (PLEG): container finished" podID="03809a9a-f12f-4df5-8b6d-73f4561c6906" containerID="3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc" exitCode=0
Sep 17 18:34:56.691892 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.691867    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"
Sep 17 18:34:56.691892 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.691886    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6" event={"ID":"03809a9a-f12f-4df5-8b6d-73f4561c6906","Type":"ContainerDied","Data":"3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc"}
Sep 17 18:34:56.692176 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.691916    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6" event={"ID":"03809a9a-f12f-4df5-8b6d-73f4561c6906","Type":"ContainerDied","Data":"3c8cf89c867b0e61714707dc17cdacb470aba6d0b454d3ad5f244e40c8947d7e"}
Sep 17 18:34:56.692176 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.691939    2589 scope.go:117] "RemoveContainer" containerID="3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc"
Sep 17 18:34:56.700538 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.700520    2589 scope.go:117] "RemoveContainer" containerID="3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc"
Sep 17 18:34:56.700809 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:34:56.700789    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc\": container with ID starting with 3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc not found: ID does not exist" containerID="3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc"
Sep 17 18:34:56.700849 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.700820    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc"} err="failed to get container status \"3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc\": rpc error: code = NotFound desc = could not find container \"3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc\": container with ID starting with 3168fd0a4a82a2c252048cf3f03ab590e21cec7f6f45e84655a26323f28dddcc not found: ID does not exist"
Sep 17 18:34:56.707288 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.707264    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"]
Sep 17 18:34:56.710083 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.710064    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-d81d7c/rate-test-model-d81d7c-kserve-777444ffc9-w4kx6"]
Sep 17 18:34:56.886123 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:56.886070    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:34:58.688147 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:58.688115    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03809a9a-f12f-4df5-8b6d-73f4561c6906" path="/var/lib/kubelet/pods/03809a9a-f12f-4df5-8b6d-73f4561c6906/volumes"
Sep 17 18:34:59.654773 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:34:59.654736    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:35:09.250598 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:09.250554    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:10.060873 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:10.060835    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:10.420620 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:10.420531    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:10.767077 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:10.767047    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:11.207915 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:11.207808    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:11.536491 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:11.536460    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:11.629381 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:11.629345    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:11.788402 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:11.788318    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:13.625195 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:13.625161    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:14.235557 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:14.235522    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:14.331350 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:14.331314    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:14.443108 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:14.443077    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:15.555752 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:15.555721    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:16.277264 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:16.277229    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.146408 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.146376    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.245575 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.245531    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.345880 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.345848    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.639779 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.639751    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.750345 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.750313    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"]
Sep 17 18:35:17.750661 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.750648    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03809a9a-f12f-4df5-8b6d-73f4561c6906" containerName="main"
Sep 17 18:35:17.750707 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.750664    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="03809a9a-f12f-4df5-8b6d-73f4561c6906" containerName="main"
Sep 17 18:35:17.750743 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.750719    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="03809a9a-f12f-4df5-8b6d-73f4561c6906" containerName="main"
Sep 17 18:35:17.753494 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.753476    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:17.756060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.756039    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-0f4423-dockercfg-sb2h4\""
Sep 17 18:35:17.756173 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.756065    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-0f4423\""
Sep 17 18:35:17.768141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.768119    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"]
Sep 17 18:35:17.858227 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.858197    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:17.906403 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.906323    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246cc06c-030b-4c25-8f41-7e65dab68931-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2\" (UID: \"246cc06c-030b-4c25-8f41-7e65dab68931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:17.906403 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:17.906366    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f45q\" (UniqueName: \"kubernetes.io/projected/246cc06c-030b-4c25-8f41-7e65dab68931-kube-api-access-4f45q\") pod \"payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2\" (UID: \"246cc06c-030b-4c25-8f41-7e65dab68931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:18.006833 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.006800    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246cc06c-030b-4c25-8f41-7e65dab68931-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2\" (UID: \"246cc06c-030b-4c25-8f41-7e65dab68931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:18.006966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.006846    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f45q\" (UniqueName: \"kubernetes.io/projected/246cc06c-030b-4c25-8f41-7e65dab68931-kube-api-access-4f45q\") pod \"payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2\" (UID: \"246cc06c-030b-4c25-8f41-7e65dab68931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:18.007551 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.007532    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246cc06c-030b-4c25-8f41-7e65dab68931-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2\" (UID: \"246cc06c-030b-4c25-8f41-7e65dab68931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:18.015096 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.015074    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f45q\" (UniqueName: \"kubernetes.io/projected/246cc06c-030b-4c25-8f41-7e65dab68931-kube-api-access-4f45q\") pod \"payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2\" (UID: \"246cc06c-030b-4c25-8f41-7e65dab68931\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:18.063065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.063036    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:18.342650 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.342624    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"]
Sep 17 18:35:18.345395 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:35:18.345366    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246cc06c_030b_4c25_8f41_7e65dab68931.slice/crio-c2a5714037ee9b332d39b39f47feaa8dc0ab48e7d5829a573cc076a7255d6ac2 WatchSource:0}: Error finding container c2a5714037ee9b332d39b39f47feaa8dc0ab48e7d5829a573cc076a7255d6ac2: Status 404 returned error can't find the container with id c2a5714037ee9b332d39b39f47feaa8dc0ab48e7d5829a573cc076a7255d6ac2
Sep 17 18:35:18.796652 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.796610    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2" event={"ID":"246cc06c-030b-4c25-8f41-7e65dab68931","Type":"ContainerStarted","Data":"ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb"}
Sep 17 18:35:18.796836 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.796676    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2" event={"ID":"246cc06c-030b-4c25-8f41-7e65dab68931","Type":"ContainerStarted","Data":"c2a5714037ee9b332d39b39f47feaa8dc0ab48e7d5829a573cc076a7255d6ac2"}
Sep 17 18:35:18.796836 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.796745    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:18.814676 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:18.814634    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2" podStartSLOduration=1.814622795 podStartE2EDuration="1.814622795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:35:18.813812556 +0000 UTC m=+1954.493498414" watchObservedRunningTime="2026-09-17 18:35:18.814622795 +0000 UTC m=+1954.494308660"
Sep 17 18:35:20.516779 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:20.516739    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:20.908380 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:20.908298    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:35:29.804618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:29.804533    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:35:38.071708 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:38.071668    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:39.334544 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:39.334510    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:39.533494 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:39.533463    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:40.270683 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:40.270655    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:40.957301 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:40.957258    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:41.850931 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:41.850894    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:42.704777 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:42.704736    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:44.617929 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:44.617892    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2_246cc06c-030b-4c25-8f41-7e65dab68931/payload-processing/0.log"
Sep 17 18:35:45.456422 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:45.456388    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:45.519500 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:45.519463    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:46.993341 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:46.993312    2589 scope.go:117] "RemoveContainer" containerID="713c2f80c725d5c2ba9305aed68485b6322c5a4beb22340aca1d350cf5e34640"
Sep 17 18:35:53.333981 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:53.333927    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:35:57.996224 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:35:57.996186    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:36:11.605137 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:11.605100    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:36:17.676392 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:17.676345    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf_d3418fd0-a73c-46b3-b119-dc7fdbc34d75/payload-processing/0.log"
Sep 17 18:36:17.922420 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:17.922384    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2_246cc06c-030b-4c25-8f41-7e65dab68931/payload-processing/0.log"
Sep 17 18:36:18.220527 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.220496    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:18.531729 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.531697    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:18.551512 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.551480    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"]
Sep 17 18:36:18.554748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.554734    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.557429 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.557406    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-c37358-openshift-default-dockercfg-462wl\""
Sep 17 18:36:18.568175 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.568149    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"]
Sep 17 18:36:18.629983 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.629956    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-certs\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.629983 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.629988    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-socket\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.630217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.630063    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cpc2\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-kube-api-access-5cpc2\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.630217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.630165    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-token\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.630217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.630194    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-data\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.630337 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.630224    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-credential-socket\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.630337 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.630282    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-envoy\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.630337 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.630319    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.630457 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.630340    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-podinfo\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.731575 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.731546    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-credential-socket\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.731589    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-envoy\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.731645    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.731674    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-podinfo\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.731728    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-certs\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.731755    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-socket\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.731798    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cpc2\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-kube-api-access-5cpc2\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.731872    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-token\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.731898    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-data\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732078 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.732036    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-credential-socket\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732546 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.732140    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-certs\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732546 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.732294    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-data\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732546 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.732335    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-socket\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.732546 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.732362    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istiod-ca-cert\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.734319 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.734298    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-envoy\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.734553 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.734535    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-podinfo\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.747379 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.747354    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-token\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.748190 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.748166    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cpc2\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-kube-api-access-5cpc2\") pod \"e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") " pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:18.851865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.851779    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:18.864659 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:18.864631    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:19.140953 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:19.140929    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"]
Sep 17 18:36:19.143417 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:36:19.143381    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51fca882_c40b_453b_b4fe_e1dd2eaaaec8.slice/crio-0df466ffe501ff85c7d2bda66e7c2f4d719f2a72444212def959bfbcc7684d7b WatchSource:0}: Error finding container 0df466ffe501ff85c7d2bda66e7c2f4d719f2a72444212def959bfbcc7684d7b: Status 404 returned error can't find the container with id 0df466ffe501ff85c7d2bda66e7c2f4d719f2a72444212def959bfbcc7684d7b
Sep 17 18:36:19.147836 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:19.147801    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:36:19.147935 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:19.147881    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:36:19.147935 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:19.147924    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:36:20.091308 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:20.091270    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" event={"ID":"51fca882-c40b-453b-b4fe-e1dd2eaaaec8","Type":"ContainerStarted","Data":"d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31"}
Sep 17 18:36:20.091679 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:20.091321    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" event={"ID":"51fca882-c40b-453b-b4fe-e1dd2eaaaec8","Type":"ContainerStarted","Data":"0df466ffe501ff85c7d2bda66e7c2f4d719f2a72444212def959bfbcc7684d7b"}
Sep 17 18:36:20.112503 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:20.112454    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" podStartSLOduration=2.112412358 podStartE2EDuration="2.112412358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:20.111644156 +0000 UTC m=+2015.791330006" watchObservedRunningTime="2026-09-17 18:36:20.112412358 +0000 UTC m=+2015.792098213"
Sep 17 18:36:20.865072 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:20.865038    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:20.870218 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:20.870195    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:21.097086 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:21.097049    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:21.098669 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:21.098647    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:21.331321 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:21.331285    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:22.064673 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:22.064639    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:22.159263 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:22.159225    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:23.147752 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:23.147711    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:24.040060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:24.040024    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:24.762390 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:24.762355    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:25.638878 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:25.638839    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:25.736692 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:25.736656    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:25.967261 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:25.967186    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:26.120065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.120033    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:26.219069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.218986    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:26.314351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.314318    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"]
Sep 17 18:36:26.317770 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.317748    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:26.320465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.320444    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-c37358-dockercfg-qth5b\""
Sep 17 18:36:26.320607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.320531    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-c37358\""
Sep 17 18:36:26.329631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.329606    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"]
Sep 17 18:36:26.399286 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.399239    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703f952-ec90-45fb-910a-cd2ec47f0af4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s\" (UID: \"e703f952-ec90-45fb-910a-cd2ec47f0af4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:26.399465 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.399371    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2nr4\" (UniqueName: \"kubernetes.io/projected/e703f952-ec90-45fb-910a-cd2ec47f0af4-kube-api-access-x2nr4\") pod \"payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s\" (UID: \"e703f952-ec90-45fb-910a-cd2ec47f0af4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:26.499966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.499877    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2nr4\" (UniqueName: \"kubernetes.io/projected/e703f952-ec90-45fb-910a-cd2ec47f0af4-kube-api-access-x2nr4\") pod \"payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s\" (UID: \"e703f952-ec90-45fb-910a-cd2ec47f0af4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:26.499966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.499957    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703f952-ec90-45fb-910a-cd2ec47f0af4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s\" (UID: \"e703f952-ec90-45fb-910a-cd2ec47f0af4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:26.500717 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.500697    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703f952-ec90-45fb-910a-cd2ec47f0af4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s\" (UID: \"e703f952-ec90-45fb-910a-cd2ec47f0af4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:26.508565 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.508540    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2nr4\" (UniqueName: \"kubernetes.io/projected/e703f952-ec90-45fb-910a-cd2ec47f0af4-kube-api-access-x2nr4\") pod \"payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s\" (UID: \"e703f952-ec90-45fb-910a-cd2ec47f0af4\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:26.629631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.629592    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:26.905551 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:26.905515    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"]
Sep 17 18:36:26.908901 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:36:26.908871    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode703f952_ec90_45fb_910a_cd2ec47f0af4.slice/crio-95ef747554bc0ca2926d34d4e8fb4a4b5c5d34e0d9dca64229fb40e3475f20d1 WatchSource:0}: Error finding container 95ef747554bc0ca2926d34d4e8fb4a4b5c5d34e0d9dca64229fb40e3475f20d1: Status 404 returned error can't find the container with id 95ef747554bc0ca2926d34d4e8fb4a4b5c5d34e0d9dca64229fb40e3475f20d1
Sep 17 18:36:27.125513 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:27.125480    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s" event={"ID":"e703f952-ec90-45fb-910a-cd2ec47f0af4","Type":"ContainerStarted","Data":"7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25"}
Sep 17 18:36:27.125736 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:27.125531    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s" event={"ID":"e703f952-ec90-45fb-910a-cd2ec47f0af4","Type":"ContainerStarted","Data":"95ef747554bc0ca2926d34d4e8fb4a4b5c5d34e0d9dca64229fb40e3475f20d1"}
Sep 17 18:36:27.125736 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:27.125599    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:27.144930 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:27.144881    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s" podStartSLOduration=1.144865143 podStartE2EDuration="1.144865143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:36:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:36:27.14248472 +0000 UTC m=+2022.822170571" watchObservedRunningTime="2026-09-17 18:36:27.144865143 +0000 UTC m=+2022.824551017"
Sep 17 18:36:31.398857 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:31.398819    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:36:38.132206 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:38.132168    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:36:42.198483 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:42.198434    2589 generic.go:356] "Generic (PLEG): container finished" podID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerID="d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31" exitCode=0
Sep 17 18:36:42.198926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:42.198506    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" event={"ID":"51fca882-c40b-453b-b4fe-e1dd2eaaaec8","Type":"ContainerDied","Data":"d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31"}
Sep 17 18:36:42.198926 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:42.198878    2589 scope.go:117] "RemoveContainer" containerID="d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31"
Sep 17 18:36:42.200050 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:42.200000    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:36:42.200109 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:42.200096    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:36:42.200147 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:42.200127    2589 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892144Ki","pods":"250"}
Sep 17 18:36:43.215401 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:43.215354    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" event={"ID":"51fca882-c40b-453b-b4fe-e1dd2eaaaec8","Type":"ContainerStarted","Data":"0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4"}
Sep 17 18:36:43.864812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:43.864774    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:43.865024 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:43.864981    2589 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 18:36:43.865087 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:43.865068    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 18:36:44.866079 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:44.866044    2589 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 18:36:44.866550 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:44.866122    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 18:36:45.865119 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:45.865075    2589 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 18:36:45.865331 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:45.865148    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 18:36:46.865881 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:46.865837    2589 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 18:36:46.866296 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:46.865909    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 18:36:47.017935 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:47.017902    2589 scope.go:117] "RemoveContainer" containerID="f454513b6466736453b8fe32300246e110d53c38ebbb32a6198e4d8f6c0840ea"
Sep 17 18:36:47.026201 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:47.026178    2589 scope.go:117] "RemoveContainer" containerID="aaadf039dd6e0135b8968f450a9e426b5402a0d7fe2ed855468d5209b79c00f7"
Sep 17 18:36:47.033721 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:47.033704    2589 scope.go:117] "RemoveContainer" containerID="fc4899852e5acbae93a9b31fe547c7d27b0a3897022d69df62f072ba1ba77f53"
Sep 17 18:36:47.040925 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:47.040902    2589 scope.go:117] "RemoveContainer" containerID="ce06a6048f2e76b5df5a95402539bbe25f3001652baf8bc1570868f16c896aae"
Sep 17 18:36:47.047971 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:47.047954    2589 scope.go:117] "RemoveContainer" containerID="80ef95599fb582471063d9835690f3ed3b886d38b51197028794e6d8d7b37e39"
Sep 17 18:36:47.728335 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:47.728298    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:47.865849 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:47.865813    2589 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Sep 17 18:36:47.866060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:47.865868    2589 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Sep 17 18:36:48.514812 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:48.514780    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:48.869792 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:48.869716    2589 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:48.870169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:48.869999    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:48.870581 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:48.870563    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:36:49.338722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:49.338692    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:50.042980 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:50.042947    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:50.725316 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:50.725287    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:51.555364 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:51.555334    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:52.172238 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:52.172204    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:53.613328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:53.613288    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:54.406139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:54.406108    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:55.246414 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:55.246376    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:55.927687 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:55.927594    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:56.840858 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:56.840815    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:36:57.586599 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:36:57.586565    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:03.955722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:03.955681    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:04.006077 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.006045    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"]
Sep 17 18:37:04.006291 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.006263    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s" podUID="e703f952-ec90-45fb-910a-cd2ec47f0af4" containerName="payload-pre-processing" containerID="cri-o://7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25" gracePeriod=30
Sep 17 18:37:04.087731 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.087693    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:04.211736 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.211647    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:04.250211 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.250185    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:37:04.268256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.268222    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x2nr4\" (UniqueName: \"kubernetes.io/projected/e703f952-ec90-45fb-910a-cd2ec47f0af4-kube-api-access-x2nr4\") pod \"e703f952-ec90-45fb-910a-cd2ec47f0af4\" (UID: \"e703f952-ec90-45fb-910a-cd2ec47f0af4\") "
Sep 17 18:37:04.268410 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.268373    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703f952-ec90-45fb-910a-cd2ec47f0af4-plugins-config-volume\") pod \"e703f952-ec90-45fb-910a-cd2ec47f0af4\" (UID: \"e703f952-ec90-45fb-910a-cd2ec47f0af4\") "
Sep 17 18:37:04.268732 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.268710    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e703f952-ec90-45fb-910a-cd2ec47f0af4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e703f952-ec90-45fb-910a-cd2ec47f0af4" (UID: "e703f952-ec90-45fb-910a-cd2ec47f0af4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:04.270564 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.270541    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e703f952-ec90-45fb-910a-cd2ec47f0af4-kube-api-access-x2nr4" (OuterVolumeSpecName: "kube-api-access-x2nr4") pod "e703f952-ec90-45fb-910a-cd2ec47f0af4" (UID: "e703f952-ec90-45fb-910a-cd2ec47f0af4"). InnerVolumeSpecName "kube-api-access-x2nr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:04.309304 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.309271    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:04.317623 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.317599    2589 generic.go:356] "Generic (PLEG): container finished" podID="e703f952-ec90-45fb-910a-cd2ec47f0af4" containerID="7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25" exitCode=0
Sep 17 18:37:04.317730 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.317639    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"
Sep 17 18:37:04.317730 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.317686    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s" event={"ID":"e703f952-ec90-45fb-910a-cd2ec47f0af4","Type":"ContainerDied","Data":"7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25"}
Sep 17 18:37:04.317804 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.317735    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s" event={"ID":"e703f952-ec90-45fb-910a-cd2ec47f0af4","Type":"ContainerDied","Data":"95ef747554bc0ca2926d34d4e8fb4a4b5c5d34e0d9dca64229fb40e3475f20d1"}
Sep 17 18:37:04.317804 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.317759    2589 scope.go:117] "RemoveContainer" containerID="7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25"
Sep 17 18:37:04.326646 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.326623    2589 scope.go:117] "RemoveContainer" containerID="7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25"
Sep 17 18:37:04.326921 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:37:04.326902    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25\": container with ID starting with 7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25 not found: ID does not exist" containerID="7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25"
Sep 17 18:37:04.326987 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.326927    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25"} err="failed to get container status \"7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25\": rpc error: code = NotFound desc = could not find container \"7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25\": container with ID starting with 7bb5247f766d956f459fa80c0b4837cdd85eee2d9b23110f69897180f4fb2c25 not found: ID does not exist"
Sep 17 18:37:04.341408 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.341385    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"]
Sep 17 18:37:04.348409 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.348385    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-c37358-59bd4c4554-wgh7s"]
Sep 17 18:37:04.374463 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.369749    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e703f952-ec90-45fb-910a-cd2ec47f0af4-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:04.374463 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.369779    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2nr4\" (UniqueName: \"kubernetes.io/projected/e703f952-ec90-45fb-910a-cd2ec47f0af4-kube-api-access-x2nr4\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:04.659580 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.659548    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:04.683226 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.683192    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e703f952-ec90-45fb-910a-cd2ec47f0af4" path="/var/lib/kubelet/pods/e703f952-ec90-45fb-910a-cd2ec47f0af4/volumes"
Sep 17 18:37:04.757217 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:04.757189    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:05.552785 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:05.552745    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:06.389950 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:06.389914    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:07.004292 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:07.004255    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:37:07.131758 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:07.131728    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:08.062671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:08.062637    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:08.279768 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:08.279738    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:10.852785 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:10.852745    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:10.953857 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:10.953810    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:11.109046 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:11.108932    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"]
Sep 17 18:37:11.109232 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:11.109207    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy" containerID="cri-o://0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4" gracePeriod=30
Sep 17 18:37:11.132466 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:11.132437    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:15.827604 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:15.827570    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:16.362395 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.362373    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:37:16.373422 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.373357    2589 generic.go:356] "Generic (PLEG): container finished" podID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerID="0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4" exitCode=0
Sep 17 18:37:16.373422 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.373401    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"
Sep 17 18:37:16.373578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.373436    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" event={"ID":"51fca882-c40b-453b-b4fe-e1dd2eaaaec8","Type":"ContainerDied","Data":"0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4"}
Sep 17 18:37:16.373578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.373491    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g" event={"ID":"51fca882-c40b-453b-b4fe-e1dd2eaaaec8","Type":"ContainerDied","Data":"0df466ffe501ff85c7d2bda66e7c2f4d719f2a72444212def959bfbcc7684d7b"}
Sep 17 18:37:16.373578 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.373517    2589 scope.go:117] "RemoveContainer" containerID="0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4"
Sep 17 18:37:16.382123 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.382093    2589 scope.go:117] "RemoveContainer" containerID="d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31"
Sep 17 18:37:16.397781 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.397760    2589 scope.go:117] "RemoveContainer" containerID="0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4"
Sep 17 18:37:16.398144 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:37:16.398123    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4\": container with ID starting with 0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4 not found: ID does not exist" containerID="0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4"
Sep 17 18:37:16.398196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.398152    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4"} err="failed to get container status \"0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4\": rpc error: code = NotFound desc = could not find container \"0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4\": container with ID starting with 0c55398ec0274965fb0499ae3464a04869b2889c5a03a6f8d2239cafa85007c4 not found: ID does not exist"
Sep 17 18:37:16.398196 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.398171    2589 scope.go:117] "RemoveContainer" containerID="d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31"
Sep 17 18:37:16.398437 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:37:16.398422    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31\": container with ID starting with d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31 not found: ID does not exist" containerID="d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31"
Sep 17 18:37:16.398491 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.398441    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31"} err="failed to get container status \"d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31\": rpc error: code = NotFound desc = could not find container \"d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31\": container with ID starting with d4da0b759392bca0286fda1222bab5b3b8dc9b0c988a2599f7f5c9633a548d31 not found: ID does not exist"
Sep 17 18:37:16.474251 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474214    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-token\") pod \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") "
Sep 17 18:37:16.474426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474267    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istiod-ca-cert\") pod \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") "
Sep 17 18:37:16.474426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474286    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-podinfo\") pod \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") "
Sep 17 18:37:16.474426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474307    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-data\") pod \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") "
Sep 17 18:37:16.474426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474356    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-envoy\") pod \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") "
Sep 17 18:37:16.474426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474373    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-certs\") pod \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") "
Sep 17 18:37:16.474426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474407    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-credential-socket\") pod \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") "
Sep 17 18:37:16.474748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474466    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5cpc2\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-kube-api-access-5cpc2\") pod \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") "
Sep 17 18:37:16.474748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474495    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-socket\") pod \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\" (UID: \"51fca882-c40b-453b-b4fe-e1dd2eaaaec8\") "
Sep 17 18:37:16.474748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474690    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-data" (OuterVolumeSpecName: "istio-data") pod "51fca882-c40b-453b-b4fe-e1dd2eaaaec8" (UID: "51fca882-c40b-453b-b4fe-e1dd2eaaaec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:16.474748 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474719    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "51fca882-c40b-453b-b4fe-e1dd2eaaaec8" (UID: "51fca882-c40b-453b-b4fe-e1dd2eaaaec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:16.474951 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474748    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "51fca882-c40b-453b-b4fe-e1dd2eaaaec8" (UID: "51fca882-c40b-453b-b4fe-e1dd2eaaaec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:16.474951 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474767    2589 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:16.474951 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474916    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "51fca882-c40b-453b-b4fe-e1dd2eaaaec8" (UID: "51fca882-c40b-453b-b4fe-e1dd2eaaaec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:16.475322 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.474992    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "51fca882-c40b-453b-b4fe-e1dd2eaaaec8" (UID: "51fca882-c40b-453b-b4fe-e1dd2eaaaec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:16.476711 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.476679    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "51fca882-c40b-453b-b4fe-e1dd2eaaaec8" (UID: "51fca882-c40b-453b-b4fe-e1dd2eaaaec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:37:16.476825 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.476741    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-token" (OuterVolumeSpecName: "istio-token") pod "51fca882-c40b-453b-b4fe-e1dd2eaaaec8" (UID: "51fca882-c40b-453b-b4fe-e1dd2eaaaec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:16.476874 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.476831    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-kube-api-access-5cpc2" (OuterVolumeSpecName: "kube-api-access-5cpc2") pod "51fca882-c40b-453b-b4fe-e1dd2eaaaec8" (UID: "51fca882-c40b-453b-b4fe-e1dd2eaaaec8"). InnerVolumeSpecName "kube-api-access-5cpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:16.476968 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.476947    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "51fca882-c40b-453b-b4fe-e1dd2eaaaec8" (UID: "51fca882-c40b-453b-b4fe-e1dd2eaaaec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:37:16.576059 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.576028    2589 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:16.576059 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.576053    2589 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:16.576059 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.576063    2589 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:16.576297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.576072    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cpc2\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-kube-api-access-5cpc2\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:16.576297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.576082    2589 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:16.576297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.576090    2589 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:16.576297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.576098    2589 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:16.576297 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.576105    2589 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51fca882-c40b-453b-b4fe-e1dd2eaaaec8-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:16.616485 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.616448    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:16.696629 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.696596    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"]
Sep 17 18:37:16.703250 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:16.703219    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-c37358-openshift-default-6d947fd594-6tw7g"]
Sep 17 18:37:17.419068 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:17.419036    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:18.159275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:18.159242    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:18.683897 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:18.683864    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" path="/var/lib/kubelet/pods/51fca882-c40b-453b-b4fe-e1dd2eaaaec8/volumes"
Sep 17 18:37:18.863388 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:18.863356    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:27.902448 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:27.902407    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:37:30.334430 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.334400    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:30.368260 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.368225    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"]
Sep 17 18:37:30.368529 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.368491    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf" podUID="d3418fd0-a73c-46b3-b119-dc7fdbc34d75" containerName="payload-processing" containerID="cri-o://53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268" gracePeriod=30
Sep 17 18:37:30.407152 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.407121    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:30.508141 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.508087    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:30.601648 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.601567    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:30.629571 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.629548    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:37:30.805415 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.805380    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54d9x\" (UniqueName: \"kubernetes.io/projected/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-kube-api-access-54d9x\") pod \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\" (UID: \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\") "
Sep 17 18:37:30.805588 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.805460    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-plugins-config-volume\") pod \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\" (UID: \"d3418fd0-a73c-46b3-b119-dc7fdbc34d75\") "
Sep 17 18:37:30.805869 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.805843    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d3418fd0-a73c-46b3-b119-dc7fdbc34d75" (UID: "d3418fd0-a73c-46b3-b119-dc7fdbc34d75"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:37:30.807722 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.807693    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-kube-api-access-54d9x" (OuterVolumeSpecName: "kube-api-access-54d9x") pod "d3418fd0-a73c-46b3-b119-dc7fdbc34d75" (UID: "d3418fd0-a73c-46b3-b119-dc7fdbc34d75"). InnerVolumeSpecName "kube-api-access-54d9x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:37:30.906532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.906425    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54d9x\" (UniqueName: \"kubernetes.io/projected/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-kube-api-access-54d9x\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:30.906532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:30.906477    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d3418fd0-a73c-46b3-b119-dc7fdbc34d75-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:37:31.000418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.000387    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:31.126034 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.125978    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:31.446818 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.446788    2589 generic.go:356] "Generic (PLEG): container finished" podID="d3418fd0-a73c-46b3-b119-dc7fdbc34d75" containerID="53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268" exitCode=0
Sep 17 18:37:31.447254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.446829    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"
Sep 17 18:37:31.447254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.446841    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf" event={"ID":"d3418fd0-a73c-46b3-b119-dc7fdbc34d75","Type":"ContainerDied","Data":"53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268"}
Sep 17 18:37:31.447254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.446884    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf" event={"ID":"d3418fd0-a73c-46b3-b119-dc7fdbc34d75","Type":"ContainerDied","Data":"f2384a20d176c15f2daf46912c2b67cc170ca32b456c1fe6e04d345159ae1cc8"}
Sep 17 18:37:31.447254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.446907    2589 scope.go:117] "RemoveContainer" containerID="53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268"
Sep 17 18:37:31.455865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.455847    2589 scope.go:117] "RemoveContainer" containerID="53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268"
Sep 17 18:37:31.456135 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:37:31.456115    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268\": container with ID starting with 53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268 not found: ID does not exist" containerID="53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268"
Sep 17 18:37:31.456219 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.456146    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268"} err="failed to get container status \"53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268\": rpc error: code = NotFound desc = could not find container \"53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268\": container with ID starting with 53fa2db8757d2645ef7e2756d060c9763b8f215dc83294afa0f959d520c5e268 not found: ID does not exist"
Sep 17 18:37:31.473346 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.473321    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"]
Sep 17 18:37:31.487429 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:31.487407    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f72155-b6c84c8fc-ws9vf"]
Sep 17 18:37:32.125743 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:32.125716    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:32.683231 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:32.683200    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3418fd0-a73c-46b3-b119-dc7fdbc34d75" path="/var/lib/kubelet/pods/d3418fd0-a73c-46b3-b119-dc7fdbc34d75/volumes"
Sep 17 18:37:33.375669 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:33.375636    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:34.224375 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:34.224344    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:36.039503 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:36.039465    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:37:41.099985 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:41.099944    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:37:46.720879 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:46.720849    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:37:46.722944 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:37:46.722917    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:38:10.229749 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:10.229714    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:12.085409 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:12.085376    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:13.537736 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:13.537707    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:17.078230 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:17.078167    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:19.566938 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:19.566897    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:20.146559 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:20.146524    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:20.256245 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:20.256208    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:26.286034 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:26.285933    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:26.494586 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:26.494551    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:26.697865 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:26.697750    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:27.011405 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011370    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"]
Sep 17 18:38:27.011776 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011758    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy"
Sep 17 18:38:27.011864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011779    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy"
Sep 17 18:38:27.011864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011791    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy"
Sep 17 18:38:27.011864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011799    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy"
Sep 17 18:38:27.011864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011813    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e703f952-ec90-45fb-910a-cd2ec47f0af4" containerName="payload-pre-processing"
Sep 17 18:38:27.011864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011823    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="e703f952-ec90-45fb-910a-cd2ec47f0af4" containerName="payload-pre-processing"
Sep 17 18:38:27.011864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011848    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3418fd0-a73c-46b3-b119-dc7fdbc34d75" containerName="payload-processing"
Sep 17 18:38:27.011864 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011857    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3418fd0-a73c-46b3-b119-dc7fdbc34d75" containerName="payload-processing"
Sep 17 18:38:27.012256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011933    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="e703f952-ec90-45fb-910a-cd2ec47f0af4" containerName="payload-pre-processing"
Sep 17 18:38:27.012256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011948    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy"
Sep 17 18:38:27.012256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011956    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="51fca882-c40b-453b-b4fe-e1dd2eaaaec8" containerName="istio-proxy"
Sep 17 18:38:27.012256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.011969    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3418fd0-a73c-46b3-b119-dc7fdbc34d75" containerName="payload-processing"
Sep 17 18:38:27.015083 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.015064    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"
Sep 17 18:38:27.022329 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.022309    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:38:27.022423 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.022379    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:38:27.037677 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.037652    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"]
Sep 17 18:38:27.191400 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.191366    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f538b5c-7574-4410-8f43-dbda088fd15f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw\" (UID: \"6f538b5c-7574-4410-8f43-dbda088fd15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"
Sep 17 18:38:27.199624 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.199592    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:27.292818 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.292741    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f538b5c-7574-4410-8f43-dbda088fd15f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw\" (UID: \"6f538b5c-7574-4410-8f43-dbda088fd15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"
Sep 17 18:38:27.293351 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.293331    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f538b5c-7574-4410-8f43-dbda088fd15f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw\" (UID: \"6f538b5c-7574-4410-8f43-dbda088fd15f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"
Sep 17 18:38:27.325712 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.325690    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"
Sep 17 18:38:27.601354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.601324    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"]
Sep 17 18:38:27.603262 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:38:27.603231    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f538b5c_7574_4410_8f43_dbda088fd15f.slice/crio-286969fb46d8e5fb028a1bd99bb4f32750452ba677a5fc158f6e9d7181ef618f WatchSource:0}: Error finding container 286969fb46d8e5fb028a1bd99bb4f32750452ba677a5fc158f6e9d7181ef618f: Status 404 returned error can't find the container with id 286969fb46d8e5fb028a1bd99bb4f32750452ba677a5fc158f6e9d7181ef618f
Sep 17 18:38:27.701530 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:27.701499    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw" event={"ID":"6f538b5c-7574-4410-8f43-dbda088fd15f","Type":"ContainerStarted","Data":"286969fb46d8e5fb028a1bd99bb4f32750452ba677a5fc158f6e9d7181ef618f"}
Sep 17 18:38:28.072084 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:28.072052    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:28.706606 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:28.706527    2589 generic.go:356] "Generic (PLEG): container finished" podID="6f538b5c-7574-4410-8f43-dbda088fd15f" containerID="4f99d3971f5a4a55b44566bd33ca733ee499188e72ba1c7d44f3e526305feb19" exitCode=0
Sep 17 18:38:28.706606 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:28.706583    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw" event={"ID":"6f538b5c-7574-4410-8f43-dbda088fd15f","Type":"ContainerDied","Data":"4f99d3971f5a4a55b44566bd33ca733ee499188e72ba1c7d44f3e526305feb19"}
Sep 17 18:38:29.350800 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:29.350764    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:29.843275 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:29.843244    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"
Sep 17 18:38:30.015491 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:30.015460    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f538b5c-7574-4410-8f43-dbda088fd15f-maas-api-service-ca\") pod \"6f538b5c-7574-4410-8f43-dbda088fd15f\" (UID: \"6f538b5c-7574-4410-8f43-dbda088fd15f\") "
Sep 17 18:38:30.015827 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:30.015805    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f538b5c-7574-4410-8f43-dbda088fd15f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6f538b5c-7574-4410-8f43-dbda088fd15f" (UID: "6f538b5c-7574-4410-8f43-dbda088fd15f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:38:30.116867 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:30.116832    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f538b5c-7574-4410-8f43-dbda088fd15f-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:38:30.196896 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:30.196858    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:30.716503 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:30.716468    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw" event={"ID":"6f538b5c-7574-4410-8f43-dbda088fd15f","Type":"ContainerDied","Data":"286969fb46d8e5fb028a1bd99bb4f32750452ba677a5fc158f6e9d7181ef618f"}
Sep 17 18:38:30.716503 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:30.716498    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"
Sep 17 18:38:30.716735 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:30.716517    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="286969fb46d8e5fb028a1bd99bb4f32750452ba677a5fc158f6e9d7181ef618f"
Sep 17 18:38:31.348966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:31.348938    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:37.961541 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:37.961509    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:38.018135 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.018104    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"]
Sep 17 18:38:38.018332 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.018310    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2" podUID="246cc06c-030b-4c25-8f41-7e65dab68931" containerName="payload-processing" containerID="cri-o://ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb" gracePeriod=30
Sep 17 18:38:38.077105 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.077069    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:38.224393 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.224324    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:38.273549 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.273526    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:38:38.289363 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.289339    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246cc06c-030b-4c25-8f41-7e65dab68931-plugins-config-volume\") pod \"246cc06c-030b-4c25-8f41-7e65dab68931\" (UID: \"246cc06c-030b-4c25-8f41-7e65dab68931\") "
Sep 17 18:38:38.289506 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.289412    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4f45q\" (UniqueName: \"kubernetes.io/projected/246cc06c-030b-4c25-8f41-7e65dab68931-kube-api-access-4f45q\") pod \"246cc06c-030b-4c25-8f41-7e65dab68931\" (UID: \"246cc06c-030b-4c25-8f41-7e65dab68931\") "
Sep 17 18:38:38.289661 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.289639    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246cc06c-030b-4c25-8f41-7e65dab68931-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "246cc06c-030b-4c25-8f41-7e65dab68931" (UID: "246cc06c-030b-4c25-8f41-7e65dab68931"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:38:38.291625 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.291602    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246cc06c-030b-4c25-8f41-7e65dab68931-kube-api-access-4f45q" (OuterVolumeSpecName: "kube-api-access-4f45q") pod "246cc06c-030b-4c25-8f41-7e65dab68931" (UID: "246cc06c-030b-4c25-8f41-7e65dab68931"). InnerVolumeSpecName "kube-api-access-4f45q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:38:38.349969 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.349929    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:38.366295 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.366253    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"]
Sep 17 18:38:38.375522 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.375493    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0f4423-dgthw"]
Sep 17 18:38:38.390298 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.390267    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/246cc06c-030b-4c25-8f41-7e65dab68931-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:38:38.390298 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.390293    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f45q\" (UniqueName: \"kubernetes.io/projected/246cc06c-030b-4c25-8f41-7e65dab68931-kube-api-access-4f45q\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:38:38.512191 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.512145    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:38.683339 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.683303    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f538b5c-7574-4410-8f43-dbda088fd15f" path="/var/lib/kubelet/pods/6f538b5c-7574-4410-8f43-dbda088fd15f/volumes"
Sep 17 18:38:38.754107 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.754060    2589 generic.go:356] "Generic (PLEG): container finished" podID="246cc06c-030b-4c25-8f41-7e65dab68931" containerID="ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb" exitCode=0
Sep 17 18:38:38.754271 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.754124    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"
Sep 17 18:38:38.754271 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.754124    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2" event={"ID":"246cc06c-030b-4c25-8f41-7e65dab68931","Type":"ContainerDied","Data":"ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb"}
Sep 17 18:38:38.754271 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.754265    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2" event={"ID":"246cc06c-030b-4c25-8f41-7e65dab68931","Type":"ContainerDied","Data":"c2a5714037ee9b332d39b39f47feaa8dc0ab48e7d5829a573cc076a7255d6ac2"}
Sep 17 18:38:38.754389 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.754295    2589 scope.go:117] "RemoveContainer" containerID="ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb"
Sep 17 18:38:38.763305 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.763284    2589 scope.go:117] "RemoveContainer" containerID="ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb"
Sep 17 18:38:38.763585 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:38:38.763562    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb\": container with ID starting with ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb not found: ID does not exist" containerID="ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb"
Sep 17 18:38:38.763644 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.763595    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb"} err="failed to get container status \"ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb\": rpc error: code = NotFound desc = could not find container \"ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb\": container with ID starting with ced507a34331fd19d2710a415b6d9966bd3caf31bd29d8049b63350bcbcdcffb not found: ID does not exist"
Sep 17 18:38:38.794349 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.794322    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"]
Sep 17 18:38:38.800822 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.800801    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-0f4423-678745d849-qlnl2"]
Sep 17 18:38:38.801319 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:38.801297    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:39.056980 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:39.056888    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:40.364033 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:40.363983    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:40.683595 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:40.683501    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="246cc06c-030b-4c25-8f41-7e65dab68931" path="/var/lib/kubelet/pods/246cc06c-030b-4c25-8f41-7e65dab68931/volumes"
Sep 17 18:38:41.579364 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:41.579329    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:41.748894 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:41.748860    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:38:45.102372 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.102309    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:45.200353 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.200321    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:45.376982 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.376897    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:45.753788 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.753760    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"]
Sep 17 18:38:45.754132 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.754117    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="246cc06c-030b-4c25-8f41-7e65dab68931" containerName="payload-processing"
Sep 17 18:38:45.754198 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.754134    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="246cc06c-030b-4c25-8f41-7e65dab68931" containerName="payload-processing"
Sep 17 18:38:45.754198 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.754157    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f538b5c-7574-4410-8f43-dbda088fd15f" containerName="revoke-keys"
Sep 17 18:38:45.754198 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.754161    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f538b5c-7574-4410-8f43-dbda088fd15f" containerName="revoke-keys"
Sep 17 18:38:45.754328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.754223    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f538b5c-7574-4410-8f43-dbda088fd15f" containerName="revoke-keys"
Sep 17 18:38:45.754328 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.754235    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="246cc06c-030b-4c25-8f41-7e65dab68931" containerName="payload-processing"
Sep 17 18:38:45.758908 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.758870    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"
Sep 17 18:38:45.761417 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.761397    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7wnm8\""
Sep 17 18:38:45.761583 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.761563    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 17 18:38:45.772614 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.772589    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"]
Sep 17 18:38:45.855323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.855291    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831e94a0-6a37-4862-98a0-7086cea33c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv\" (UID: \"831e94a0-6a37-4862-98a0-7086cea33c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"
Sep 17 18:38:45.883067 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.883034    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:45.956025 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.955970    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831e94a0-6a37-4862-98a0-7086cea33c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv\" (UID: \"831e94a0-6a37-4862-98a0-7086cea33c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"
Sep 17 18:38:45.956558 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:45.956539    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831e94a0-6a37-4862-98a0-7086cea33c87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv\" (UID: \"831e94a0-6a37-4862-98a0-7086cea33c87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"
Sep 17 18:38:46.071065 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:46.070951    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"
Sep 17 18:38:46.341375 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:46.341352    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"]
Sep 17 18:38:46.344316 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:38:46.344283    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod831e94a0_6a37_4862_98a0_7086cea33c87.slice/crio-a6d96d67c5db29b01f6d42959b83efb8e020668a3921b011a8066aa1ae786495 WatchSource:0}: Error finding container a6d96d67c5db29b01f6d42959b83efb8e020668a3921b011a8066aa1ae786495: Status 404 returned error can't find the container with id a6d96d67c5db29b01f6d42959b83efb8e020668a3921b011a8066aa1ae786495
Sep 17 18:38:46.795023 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:46.794980    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv" event={"ID":"831e94a0-6a37-4862-98a0-7086cea33c87","Type":"ContainerStarted","Data":"f60322075861bcab44286e3d1d7d2bbce15827034cc11844ebb559879c234e39"}
Sep 17 18:38:46.795197 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:46.795051    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv" event={"ID":"831e94a0-6a37-4862-98a0-7086cea33c87","Type":"ContainerStarted","Data":"a6d96d67c5db29b01f6d42959b83efb8e020668a3921b011a8066aa1ae786495"}
Sep 17 18:38:46.819152 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:46.819106    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv" podStartSLOduration=1.819093756 podStartE2EDuration="1.819093756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-17 18:38:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:38:46.817816805 +0000 UTC m=+2162.497502659" watchObservedRunningTime="2026-09-17 18:38:46.819093756 +0000 UTC m=+2162.498779610"
Sep 17 18:38:46.969949 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:46.969856    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:47.801136 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:47.801100    2589 generic.go:356] "Generic (PLEG): container finished" podID="831e94a0-6a37-4862-98a0-7086cea33c87" containerID="f60322075861bcab44286e3d1d7d2bbce15827034cc11844ebb559879c234e39" exitCode=0
Sep 17 18:38:47.801563 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:47.801185    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv" event={"ID":"831e94a0-6a37-4862-98a0-7086cea33c87","Type":"ContainerDied","Data":"f60322075861bcab44286e3d1d7d2bbce15827034cc11844ebb559879c234e39"}
Sep 17 18:38:48.034546 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:48.034507    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:48.822814 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:48.822778    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:48.938746 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:48.938724    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"
Sep 17 18:38:48.982256 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:48.982231    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831e94a0-6a37-4862-98a0-7086cea33c87-maas-api-service-ca\") pod \"831e94a0-6a37-4862-98a0-7086cea33c87\" (UID: \"831e94a0-6a37-4862-98a0-7086cea33c87\") "
Sep 17 18:38:48.982609 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:48.982585    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/831e94a0-6a37-4862-98a0-7086cea33c87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "831e94a0-6a37-4862-98a0-7086cea33c87" (UID: "831e94a0-6a37-4862-98a0-7086cea33c87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:38:49.083129 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:49.083049    2589 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/831e94a0-6a37-4862-98a0-7086cea33c87-maas-api-service-ca\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:38:49.810842 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:49.810810    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"
Sep 17 18:38:49.811057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:49.810837    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv" event={"ID":"831e94a0-6a37-4862-98a0-7086cea33c87","Type":"ContainerDied","Data":"a6d96d67c5db29b01f6d42959b83efb8e020668a3921b011a8066aa1ae786495"}
Sep 17 18:38:49.811057 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:49.810885    2589 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6d96d67c5db29b01f6d42959b83efb8e020668a3921b011a8066aa1ae786495"
Sep 17 18:38:56.599951 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:56.599902    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:56.698983 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:56.698952    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:56.857070 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:56.856962    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:56.876791 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:56.876759    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"]
Sep 17 18:38:56.885903 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:56.885874    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-95a92b-lrldv"]
Sep 17 18:38:56.966783 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:56.966751    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:57.334929 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:57.334890    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:57.528056 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:57.528018    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:58.611028 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:58.610976    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:38:58.683253 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:58.683220    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="831e94a0-6a37-4862-98a0-7086cea33c87" path="/var/lib/kubelet/pods/831e94a0-6a37-4862-98a0-7086cea33c87/volumes"
Sep 17 18:38:59.814673 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:38:59.814635    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:39:04.038426 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:04.038391    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:04.141477 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:04.141443    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:04.384880 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:04.384798    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"]
Sep 17 18:39:04.385104 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:04.385077    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy" containerID="cri-o://be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a" gracePeriod=30
Sep 17 18:39:04.401346 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:04.401316    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:05.855768 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:05.855731    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:06.757476 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:06.757424    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:08.872443 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:08.872408    2589 patch_prober.go:28] interesting pod/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 17 18:39:08.872843 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:08.872477    2589 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 17 18:39:09.633710 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.633686    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:39:09.761455 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761432    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-certs\") pod \"761b7aa1-f9cb-476a-8943-764e98cb31ac\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") "
Sep 17 18:39:09.761618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761468    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-data\") pod \"761b7aa1-f9cb-476a-8943-764e98cb31ac\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") "
Sep 17 18:39:09.761618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761486    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-credential-socket\") pod \"761b7aa1-f9cb-476a-8943-764e98cb31ac\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") "
Sep 17 18:39:09.761618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761506    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761b7aa1-f9cb-476a-8943-764e98cb31ac-istiod-ca-cert\") pod \"761b7aa1-f9cb-476a-8943-764e98cb31ac\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") "
Sep 17 18:39:09.761618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761544    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-token\") pod \"761b7aa1-f9cb-476a-8943-764e98cb31ac\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") "
Sep 17 18:39:09.761618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761566    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-socket\") pod \"761b7aa1-f9cb-476a-8943-764e98cb31ac\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") "
Sep 17 18:39:09.761618 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761601    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tzmb8\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-kube-api-access-tzmb8\") pod \"761b7aa1-f9cb-476a-8943-764e98cb31ac\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") "
Sep 17 18:39:09.761954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761635    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-envoy\") pod \"761b7aa1-f9cb-476a-8943-764e98cb31ac\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") "
Sep 17 18:39:09.761954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761664    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-podinfo\") pod \"761b7aa1-f9cb-476a-8943-764e98cb31ac\" (UID: \"761b7aa1-f9cb-476a-8943-764e98cb31ac\") "
Sep 17 18:39:09.761954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761734    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-data" (OuterVolumeSpecName: "istio-data") pod "761b7aa1-f9cb-476a-8943-764e98cb31ac" (UID: "761b7aa1-f9cb-476a-8943-764e98cb31ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:09.761954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761751    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "761b7aa1-f9cb-476a-8943-764e98cb31ac" (UID: "761b7aa1-f9cb-476a-8943-764e98cb31ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:09.761954 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.761801    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "761b7aa1-f9cb-476a-8943-764e98cb31ac" (UID: "761b7aa1-f9cb-476a-8943-764e98cb31ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:09.762254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.762033    2589 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-certs\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:09.762254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.762057    2589 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-data\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:09.762254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.762070    2589 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-credential-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:09.762254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.762130    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "761b7aa1-f9cb-476a-8943-764e98cb31ac" (UID: "761b7aa1-f9cb-476a-8943-764e98cb31ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:09.762254 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.762155    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761b7aa1-f9cb-476a-8943-764e98cb31ac-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "761b7aa1-f9cb-476a-8943-764e98cb31ac" (UID: "761b7aa1-f9cb-476a-8943-764e98cb31ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:39:09.764194 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.764164    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "761b7aa1-f9cb-476a-8943-764e98cb31ac" (UID: "761b7aa1-f9cb-476a-8943-764e98cb31ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 17 18:39:09.764323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.764184    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-kube-api-access-tzmb8" (OuterVolumeSpecName: "kube-api-access-tzmb8") pod "761b7aa1-f9cb-476a-8943-764e98cb31ac" (UID: "761b7aa1-f9cb-476a-8943-764e98cb31ac"). InnerVolumeSpecName "kube-api-access-tzmb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:09.764323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.764223    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "761b7aa1-f9cb-476a-8943-764e98cb31ac" (UID: "761b7aa1-f9cb-476a-8943-764e98cb31ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 17 18:39:09.764439 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.764359    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-token" (OuterVolumeSpecName: "istio-token") pod "761b7aa1-f9cb-476a-8943-764e98cb31ac" (UID: "761b7aa1-f9cb-476a-8943-764e98cb31ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:09.863300 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.863262    2589 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/761b7aa1-f9cb-476a-8943-764e98cb31ac-istiod-ca-cert\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:09.863300 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.863291    2589 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-token\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:09.863300 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.863300    2589 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-workload-socket\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:09.863300 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.863308    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzmb8\" (UniqueName: \"kubernetes.io/projected/761b7aa1-f9cb-476a-8943-764e98cb31ac-kube-api-access-tzmb8\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:09.863569 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.863316    2589 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-envoy\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:09.863569 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.863326    2589 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/761b7aa1-f9cb-476a-8943-764e98cb31ac-istio-podinfo\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:09.904633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.904602    2589 generic.go:356] "Generic (PLEG): container finished" podID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerID="be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a" exitCode=0
Sep 17 18:39:09.904988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.904653    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"
Sep 17 18:39:09.904988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.904691    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" event={"ID":"761b7aa1-f9cb-476a-8943-764e98cb31ac","Type":"ContainerDied","Data":"be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a"}
Sep 17 18:39:09.904988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.904742    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d" event={"ID":"761b7aa1-f9cb-476a-8943-764e98cb31ac","Type":"ContainerDied","Data":"b9335243997c06230c7431e862a7077565ef50404520b8cb90b19e2ff8d4eb1d"}
Sep 17 18:39:09.904988 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.904763    2589 scope.go:117] "RemoveContainer" containerID="be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a"
Sep 17 18:39:09.914049 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.914030    2589 scope.go:117] "RemoveContainer" containerID="2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979"
Sep 17 18:39:09.921608 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.921590    2589 scope.go:117] "RemoveContainer" containerID="be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a"
Sep 17 18:39:09.921861 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:39:09.921843    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a\": container with ID starting with be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a not found: ID does not exist" containerID="be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a"
Sep 17 18:39:09.921928 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.921869    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a"} err="failed to get container status \"be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a\": rpc error: code = NotFound desc = could not find container \"be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a\": container with ID starting with be53988d33324e0d2ee9865286573c08cba49903f12f14684b19c99437c5939a not found: ID does not exist"
Sep 17 18:39:09.921928 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.921885    2589 scope.go:117] "RemoveContainer" containerID="2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979"
Sep 17 18:39:09.922140 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:39:09.922125    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979\": container with ID starting with 2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979 not found: ID does not exist" containerID="2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979"
Sep 17 18:39:09.922188 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.922144    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979"} err="failed to get container status \"2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979\": rpc error: code = NotFound desc = could not find container \"2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979\": container with ID starting with 2640ede45dfd93be512ef7766f09313cf2e5265d7b0467b2764b009afc59a979 not found: ID does not exist"
Sep 17 18:39:09.948406 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.948384    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"]
Sep 17 18:39:09.953791 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:09.953770    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-95a92b-openshift-default-5dbf44ffb8-vq62d"]
Sep 17 18:39:10.683983 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:10.683948    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" path="/var/lib/kubelet/pods/761b7aa1-f9cb-476a-8943-764e98cb31ac/volumes"
Sep 17 18:39:15.345105 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.345071    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf"]
Sep 17 18:39:15.352229 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.352207    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-d81d7c-29827830-lpnmf"]
Sep 17 18:39:15.355395 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.355369    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:15.388474 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.388447    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"]
Sep 17 18:39:15.388690 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.388667    2589 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l" podUID="ba3dd22b-a357-4b33-a1a6-996d9121ac25" containerName="payload-pre-processing" containerID="cri-o://31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03" gracePeriod=30
Sep 17 18:39:15.428418 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.428376    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:15.528966 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.528929    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:15.611579 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.611507    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:15.642532 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.642509    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:39:15.680752 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.680716    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:15.811711 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.811680    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3dd22b-a357-4b33-a1a6-996d9121ac25-plugins-config-volume\") pod \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\" (UID: \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\") "
Sep 17 18:39:15.811898 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.811741    2589 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbmjm\" (UniqueName: \"kubernetes.io/projected/ba3dd22b-a357-4b33-a1a6-996d9121ac25-kube-api-access-tbmjm\") pod \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\" (UID: \"ba3dd22b-a357-4b33-a1a6-996d9121ac25\") "
Sep 17 18:39:15.812175 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.812139    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba3dd22b-a357-4b33-a1a6-996d9121ac25-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ba3dd22b-a357-4b33-a1a6-996d9121ac25" (UID: "ba3dd22b-a357-4b33-a1a6-996d9121ac25"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 17 18:39:15.814015 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.813984    2589 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3dd22b-a357-4b33-a1a6-996d9121ac25-kube-api-access-tbmjm" (OuterVolumeSpecName: "kube-api-access-tbmjm") pod "ba3dd22b-a357-4b33-a1a6-996d9121ac25" (UID: "ba3dd22b-a357-4b33-a1a6-996d9121ac25"). InnerVolumeSpecName "kube-api-access-tbmjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 17 18:39:15.912840 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.912762    2589 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ba3dd22b-a357-4b33-a1a6-996d9121ac25-plugins-config-volume\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:15.912840 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.912791    2589 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbmjm\" (UniqueName: \"kubernetes.io/projected/ba3dd22b-a357-4b33-a1a6-996d9121ac25-kube-api-access-tbmjm\") on node \"ip-10-0-142-105.ec2.internal\" DevicePath \"\""
Sep 17 18:39:15.936879 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.936846    2589 generic.go:356] "Generic (PLEG): container finished" podID="ba3dd22b-a357-4b33-a1a6-996d9121ac25" containerID="31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03" exitCode=0
Sep 17 18:39:15.937058 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.936884    2589 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"
Sep 17 18:39:15.937058 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.936936    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l" event={"ID":"ba3dd22b-a357-4b33-a1a6-996d9121ac25","Type":"ContainerDied","Data":"31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03"}
Sep 17 18:39:15.937058 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.936991    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l" event={"ID":"ba3dd22b-a357-4b33-a1a6-996d9121ac25","Type":"ContainerDied","Data":"380b0482af583efc601f03f80f7c3455e0b5d78fb3b145316a336675962b4df9"}
Sep 17 18:39:15.937058 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.937046    2589 scope.go:117] "RemoveContainer" containerID="31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03"
Sep 17 18:39:15.946152 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.946135    2589 scope.go:117] "RemoveContainer" containerID="31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03"
Sep 17 18:39:15.946390 ip-10-0-142-105 kubenswrapper[2589]: E0917 18:39:15.946373    2589 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03\": container with ID starting with 31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03 not found: ID does not exist" containerID="31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03"
Sep 17 18:39:15.946445 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.946397    2589 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03"} err="failed to get container status \"31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03\": rpc error: code = NotFound desc = could not find container \"31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03\": container with ID starting with 31f497bb55818d100201f178456aef87d5d8d3291f54fbdf6cd350bc8249be03 not found: ID does not exist"
Sep 17 18:39:15.958845 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.958823    2589 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"]
Sep 17 18:39:15.965440 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.965418    2589 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d81d7c-865596897d-tnr2l"]
Sep 17 18:39:15.986577 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:15.986549    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:16.088755 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:16.088708    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:16.683182 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:16.683149    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f413b49-e915-44ca-a269-10b219749a68" path="/var/lib/kubelet/pods/2f413b49-e915-44ca-a269-10b219749a68/volumes"
Sep 17 18:39:16.683553 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:16.683507    2589 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba3dd22b-a357-4b33-a1a6-996d9121ac25" path="/var/lib/kubelet/pods/ba3dd22b-a357-4b33-a1a6-996d9121ac25/volumes"
Sep 17 18:39:18.217898 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:18.217861    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:39:22.667114 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:22.667076    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:22.767359 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:22.767327    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:22.945552 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:22.945469    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:23.045189 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:23.045150    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:39:29.114970 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:29.114929    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:39:47.108022 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:39:47.107970    2589 scope.go:117] "RemoveContainer" containerID="d7614fa1148a67b452bc5ee26a9626fc4d491ff76b83fd4e04707a7237e32743"
Sep 17 18:40:52.917775 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:40:52.917739    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:40:53.569658 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:40:53.569612    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:40:55.049022 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:40:55.048942    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:06.081544 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:41:06.081466    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:06.182292 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:41:06.182254    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:28.950908 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:41:28.950802    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:32.610184 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:41:32.610146    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:41:36.921918 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:41:36.921884    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:41:40.747171 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:41:40.747135    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:41:48.895651 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:41:48.895615    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:42:05.001931 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:42:05.001897    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:42:10.905268 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:42:10.905227    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:42:31.521472 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:42:31.521430    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:42:46.747671 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:42:46.747641    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:42:46.749859 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:42:46.749835    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-m4trh_3ccf6be4-8960-4d93-b2eb-fce0f02e5930/ovn-acl-logging/0.log"
Sep 17 18:42:48.653499 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:42:48.653448    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:07.930337 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:43:07.930230    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:29.764841 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:43:29.764811    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:43:43.412102 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:43:43.412068    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:43:52.195182 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:43:52.195141    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:44:06.756985 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:44:06.756951    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:09.412512 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:44:09.412475    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:44:20.128710 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:44:20.128678    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:23.107996 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:44:23.107960    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:44:47.161428 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:44:47.161398    2589 scope.go:117] "RemoveContainer" containerID="f60322075861bcab44286e3d1d7d2bbce15827034cc11844ebb559879c234e39"
Sep 17 18:44:47.169416 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:44:47.169396    2589 scope.go:117] "RemoveContainer" containerID="4f99d3971f5a4a55b44566bd33ca733ee499188e72ba1c7d44f3e526305feb19"
Sep 17 18:44:53.152806 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:44:53.152758    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:54.263997 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:44:54.263957    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:44:56.102329 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:44:56.102289    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:45:03.931692 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:03.931659    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:45:10.996799 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:10.996753    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:45:18.895156 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:18.895124    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:45:25.700379 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:25.700343    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:45:31.700337 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:31.700299    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:45:38.700624 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:38.700589    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-v7sbj"]
Sep 17 18:45:41.080074 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:41.080048    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-697d77b6c5-976hh_5d91f7c4-3f84-4f6d-bc31-941201484f7c/authorino/0.log"
Sep 17 18:45:47.464660 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:47.464630    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-hrrfd_97efffa8-eaa8-467a-afe1-a1ac0a849718/copy-manifests/0.log"
Sep 17 18:45:47.471877 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:47.471852    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-hrrfd_97efffa8-eaa8-467a-afe1-a1ac0a849718/manager/0.log"
Sep 17 18:45:48.151665 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:48.151638    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29827830-bv6xl_cbabc825-f2e7-4ee7-80ec-207341817f7c/cleanup/0.log"
Sep 17 18:45:49.142171 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:49.142140    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-697d77b6c5-976hh_5d91f7c4-3f84-4f6d-bc31-941201484f7c/authorino/0.log"
Sep 17 18:45:49.483607 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:49.483530    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-n2mnc_6539508f-a67f-4f99-92c5-9b8492b28616/kuadrant-console-plugin/0.log"
Sep 17 18:45:49.736354 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:49.736279    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-v7sbj_be775eda-0f2f-4b43-be7d-0dcf19fe0bb4/limitador/0.log"
Sep 17 18:45:50.447678 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:50.447647    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bc8cbff6f-6vkds_a0f5e960-fac5-4865-a213-2419bf880c14/kube-auth-proxy/0.log"
Sep 17 18:45:51.751841 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:51.751812    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-ggnnw_96f9bf13-92f4-42e9-b91f-48702dd6a833/main/0.log"
Sep 17 18:45:51.860582 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:51.860554    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f78gd5_a437c422-6d2a-4493-9265-dab5e2e510ba/main/0.log"
Sep 17 18:45:51.972994 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:51.972966    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-w7mt2_a601e0a9-6c9f-4970-ace4-a20b1c8edf7b/main/0.log"
Sep 17 18:45:52.090233 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:52.090199    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-n8795_b80d2928-2b56-4a0c-b295-ac9de7190d22/main/0.log"
Sep 17 18:45:58.045873 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:58.045793    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:45:59.262940 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:45:59.262908    2589 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log" limit=1024
Sep 17 18:46:01.018866 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:01.018841    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-89zbw_48bde41b-d00e-4bfb-8fec-0b4edbbca4f1/global-pull-secret-syncer/0.log"
Sep 17 18:46:01.197951 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:01.197925    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6bf5d_c909fa0f-361f-4896-9cf5-9b20f2e8ec94/konnectivity-agent/0.log"
Sep 17 18:46:01.361069 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:01.360949    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-142-105.ec2.internal_3ff6991f08a902cdf413530d4bdd103e/haproxy/0.log"
Sep 17 18:46:06.199932 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:06.199892    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-697d77b6c5-976hh_5d91f7c4-3f84-4f6d-bc31-941201484f7c/authorino/0.log"
Sep 17 18:46:06.345070 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:06.345045    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-n2mnc_6539508f-a67f-4f99-92c5-9b8492b28616/kuadrant-console-plugin/0.log"
Sep 17 18:46:06.451303 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:06.451228    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-v7sbj_be775eda-0f2f-4b43-be7d-0dcf19fe0bb4/limitador/0.log"
Sep 17 18:46:08.587596 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:08.587564    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hqlpd_46e3b407-8f77-463c-a7bd-b12c146dd498/node-exporter/0.log"
Sep 17 18:46:08.618444 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:08.618422    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hqlpd_46e3b407-8f77-463c-a7bd-b12c146dd498/kube-rbac-proxy/0.log"
Sep 17 18:46:08.653374 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:08.653353    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hqlpd_46e3b407-8f77-463c-a7bd-b12c146dd498/init-textfile/0.log"
Sep 17 18:46:08.909191 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:08.909103    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-hb4k5_7cc5f99a-a7a9-4949-81f9-3572052afd7f/kube-rbac-proxy-main/0.log"
Sep 17 18:46:08.938848 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:08.938822    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-hb4k5_7cc5f99a-a7a9-4949-81f9-3572052afd7f/kube-rbac-proxy-self/0.log"
Sep 17 18:46:08.966606 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:08.966580    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-hb4k5_7cc5f99a-a7a9-4949-81f9-3572052afd7f/openshift-state-metrics/0.log"
Sep 17 18:46:10.828515 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:10.828485    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-fpbjm_9f49e94b-b01e-4085-940f-f2048dd17044/networking-console-plugin/0.log"
Sep 17 18:46:11.266728 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.266696    2589 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"]
Sep 17 18:46:11.267041 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267028    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba3dd22b-a357-4b33-a1a6-996d9121ac25" containerName="payload-pre-processing"
Sep 17 18:46:11.267092 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267043    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba3dd22b-a357-4b33-a1a6-996d9121ac25" containerName="payload-pre-processing"
Sep 17 18:46:11.267092 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267064    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy"
Sep 17 18:46:11.267092 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267073    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy"
Sep 17 18:46:11.267092 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267081    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy"
Sep 17 18:46:11.267092 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267087    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy"
Sep 17 18:46:11.267249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267102    2589 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="831e94a0-6a37-4862-98a0-7086cea33c87" containerName="revoke-keys"
Sep 17 18:46:11.267249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267108    2589 state_mem.go:107] "Deleted CPUSet assignment" podUID="831e94a0-6a37-4862-98a0-7086cea33c87" containerName="revoke-keys"
Sep 17 18:46:11.267249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267165    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba3dd22b-a357-4b33-a1a6-996d9121ac25" containerName="payload-pre-processing"
Sep 17 18:46:11.267249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267173    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy"
Sep 17 18:46:11.267249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267179    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="831e94a0-6a37-4862-98a0-7086cea33c87" containerName="revoke-keys"
Sep 17 18:46:11.267249 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.267184    2589 memory_manager.go:368] "RemoveStaleState removing state" podUID="761b7aa1-f9cb-476a-8943-764e98cb31ac" containerName="istio-proxy"
Sep 17 18:46:11.269921 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.269906    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.272502 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.272478    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vmxpt\"/\"kube-root-ca.crt\""
Sep 17 18:46:11.272649 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.272542    2589 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vmxpt\"/\"default-dockercfg-pgspx\""
Sep 17 18:46:11.272649 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.272540    2589 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vmxpt\"/\"openshift-service-ca.crt\""
Sep 17 18:46:11.279193 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.279172    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"]
Sep 17 18:46:11.394169 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.394135    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchxq\" (UniqueName: \"kubernetes.io/projected/011be655-9ae6-4f3c-9b48-ff913fb68b8a-kube-api-access-fchxq\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.394340 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.394183    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-podres\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.394340 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.394205    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-proc\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.394340 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.394271    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-lib-modules\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.394340 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.394314    2589 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-sys\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.495412 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.495377    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-lib-modules\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.495570 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.495419    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-sys\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.495570 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.495493    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchxq\" (UniqueName: \"kubernetes.io/projected/011be655-9ae6-4f3c-9b48-ff913fb68b8a-kube-api-access-fchxq\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.495570 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.495561    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-podres\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.495682 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.495576    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-lib-modules\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.495682 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.495586    2589 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-proc\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.495682 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.495499    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-sys\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.495682 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.495639    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-proc\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.495682 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.495664    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/011be655-9ae6-4f3c-9b48-ff913fb68b8a-podres\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.504236 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.504210    2589 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchxq\" (UniqueName: \"kubernetes.io/projected/011be655-9ae6-4f3c-9b48-ff913fb68b8a-kube-api-access-fchxq\") pod \"perf-node-gather-daemonset-njkpb\" (UID: \"011be655-9ae6-4f3c-9b48-ff913fb68b8a\") " pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.580816 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.580750    2589 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:11.849705 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:11.849681    2589 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"]
Sep 17 18:46:11.852226 ip-10-0-142-105 kubenswrapper[2589]: W0917 18:46:11.852199    2589 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod011be655_9ae6_4f3c_9b48_ff913fb68b8a.slice/crio-1ec08adb5e1aec827fa481992a8ca9456047b95c6bcba21192c3366b5dbd95db WatchSource:0}: Error finding container 1ec08adb5e1aec827fa481992a8ca9456047b95c6bcba21192c3366b5dbd95db: Status 404 returned error can't find the container with id 1ec08adb5e1aec827fa481992a8ca9456047b95c6bcba21192c3366b5dbd95db
Sep 17 18:46:12.733323 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:12.733283    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb" event={"ID":"011be655-9ae6-4f3c-9b48-ff913fb68b8a","Type":"ContainerStarted","Data":"8191c8f79416f347ab5189083e2eb92861f97c755da598eb888d9a7160e6f5be"}
Sep 17 18:46:12.733524 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:12.733335    2589 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb" event={"ID":"011be655-9ae6-4f3c-9b48-ff913fb68b8a","Type":"ContainerStarted","Data":"1ec08adb5e1aec827fa481992a8ca9456047b95c6bcba21192c3366b5dbd95db"}
Sep 17 18:46:12.733524 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:12.733395    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:12.750611 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:12.750561    2589 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb" podStartSLOduration=1.750546393 podStartE2EDuration="1.750546393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-17 18:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-17 18:46:12.748178974 +0000 UTC m=+2608.427864828" watchObservedRunningTime="2026-09-17 18:46:12.750546393 +0000 UTC m=+2608.430232246"
Sep 17 18:46:14.309728 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:14.309691    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-clrtz_fc1f8942-acd9-43aa-a7e4-e5cccc1d0359/dns/0.log"
Sep 17 18:46:14.334651 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:14.334627    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-clrtz_fc1f8942-acd9-43aa-a7e4-e5cccc1d0359/kube-rbac-proxy/0.log"
Sep 17 18:46:14.431370 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:14.431340    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-xlknw_40eaaa5e-fe21-44f7-a769-2778ab800d3f/dns-node-resolver/0.log"
Sep 17 18:46:15.132958 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:15.132931    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rb4ph_2e8325d4-12ab-4d44-9353-9853c42337ae/node-ca/0.log"
Sep 17 18:46:16.602900 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:16.602865    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bc8cbff6f-6vkds_a0f5e960-fac5-4865-a213-2419bf880c14/kube-auth-proxy/0.log"
Sep 17 18:46:17.791631 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:17.791601    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mmxsl_b1c403c8-5032-4a13-9413-cc9bc512fd3f/serve-healthcheck-canary/0.log"
Sep 17 18:46:18.479139 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:18.479111    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xq4s_9f032f75-5f73-4847-889d-98d6e9b06785/kube-rbac-proxy/0.log"
Sep 17 18:46:18.500910 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:18.500881    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xq4s_9f032f75-5f73-4847-889d-98d6e9b06785/exporter/0.log"
Sep 17 18:46:18.525741 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:18.525716    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-5xq4s_9f032f75-5f73-4847-889d-98d6e9b06785/extractor/0.log"
Sep 17 18:46:18.747726 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:18.747657    2589 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vmxpt/perf-node-gather-daemonset-njkpb"
Sep 17 18:46:21.913157 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:21.913076    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-hrrfd_97efffa8-eaa8-467a-afe1-a1ac0a849718/manager/0.log"
Sep 17 18:46:21.936060 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:21.936036    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-648c847c44-hrrfd_97efffa8-eaa8-467a-afe1-a1ac0a849718/copy-manifests/0.log"
Sep 17 18:46:22.809633 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:22.809602    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bc8cbff6f-6vkds_a0f5e960-fac5-4865-a213-2419bf880c14/kube-auth-proxy/0.log"
Sep 17 18:46:23.282429 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:23.282388    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-b8b8795dc-jhzhb_9860d101-be11-48fa-a3fc-061808d4ef2f/manager/0.log"
Sep 17 18:46:24.879382 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:24.879344    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bc8cbff6f-6vkds_a0f5e960-fac5-4865-a213-2419bf880c14/kube-auth-proxy/0.log"
Sep 17 18:46:28.523785 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:28.523752    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-xwv7z_dd39f199-8168-4a5e-b58f-4b3d7f11aace/migrator/0.log"
Sep 17 18:46:28.545924 ip-10-0-142-105 kubenswrapper[2589]: I0917 18:46:28.545901    2589 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-xwv7z_dd39f199-8168-4a5e-b58f-4b3d7f11aace/graceful-termination/0.log"