Sep 09 20:38:39.469850 ip-10-0-137-187 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 09 20:38:39.470332 ip-10-0-137-187 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 09 20:38:39.470452 ip-10-0-137-187 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 09 20:38:39.471009 ip-10-0-137-187 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 09 20:38:49.202753 ip-10-0-137-187 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 09 20:38:49.202769 ip-10-0-137-187 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot dc32bcd07aa94262941046acdfb50f22 --
Sep 09 20:40:18.196841 ip-10-0-137-187 systemd[1]: Starting Kubernetes Kubelet...
Sep 09 20:40:18.641886 ip-10-0-137-187 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:40:18.641886 ip-10-0-137-187 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 09 20:40:18.641886 ip-10-0-137-187 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:40:18.641886 ip-10-0-137-187 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 09 20:40:18.641886 ip-10-0-137-187 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 09 20:40:18.643147 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.643040    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 09 20:40:18.647169 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647155    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:40:18.647169 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647171    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647175    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647178    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647181    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647183    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647186    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647189    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647192    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647195    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647199    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647203    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647206    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647216    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647219    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647222    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647225    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647227    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647230    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647232    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:40:18.647233 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647235    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647238    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647242    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647244    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647247    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647250    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647254    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647257    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647260    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647263    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647266    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647268    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647271    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647274    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647276    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647279    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647281    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647284    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647286    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647289    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:40:18.647713 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647292    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647294    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647297    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647299    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647302    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647305    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647308    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647311    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647314    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647316    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647319    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647328    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647331    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647333    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647337    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647339    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647343    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647345    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647348    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647351    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:40:18.648194 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647353    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647356    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647358    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647361    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647365    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647368    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647370    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647372    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647375    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647378    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647380    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647384    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647387    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647389    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647392    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647396    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647398    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647401    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647403    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647406    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:40:18.648697 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647409    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647411    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647414    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647416    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647419    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647422    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647424    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647427    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647429    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647432    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647434    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647437    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647439    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647442    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647444    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647447    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647449    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647452    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.647454    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648135    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:40:18.649264 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648142    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648145    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648147    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648150    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648154    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648157    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648159    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648162    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648165    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648167    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648170    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648173    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648176    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648178    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648181    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648183    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648186    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648188    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648191    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648193    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:40:18.649798 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648196    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648199    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648201    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648204    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648207    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648209    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648212    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648214    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648217    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648219    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648222    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648225    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648227    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648230    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648233    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648235    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648238    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648241    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648244    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648246    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:40:18.650301 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648249    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648252    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648254    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648257    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648260    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648263    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648265    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648268    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648270    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648273    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648275    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648278    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648281    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648283    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648285    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648288    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648291    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648293    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648296    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648299    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:40:18.650804 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648301    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648304    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648306    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648309    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648313    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648316    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648319    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648322    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648326    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648331    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648334    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648337    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648340    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648344    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648347    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648350    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648352    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648354    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648357    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:40:18.651291 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648360    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648362    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648365    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648367    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648370    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648373    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648375    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648378    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648381    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648383    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648386    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648388    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648391    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648394    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648396    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648399    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648401    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648404    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648407    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:40:18.651799 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648770    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648775    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648779    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648782    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648786    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648789    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648792    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648795    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648798    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648800    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648803    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648805    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648808    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648811    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648813    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648816    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648819    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648822    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648825    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648827    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:40:18.652270 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648830    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648833    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648835    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648839    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648843    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648846    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648849    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648852    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648855    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648857    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648860    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648862    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648865    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648868    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648870    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648872    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648875    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648878    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648880    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648883    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:40:18.652763 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648886    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648888    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648891    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648893    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648896    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648898    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648901    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648904    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648906    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648909    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648911    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648914    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648919    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648921    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648924    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648926    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648929    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648932    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648934    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:40:18.653239 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648937    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648939    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648942    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648944    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648947    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648949    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648952    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648954    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648957    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648959    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648961    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648966    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648969    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648972    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648975    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648978    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648981    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648984    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648986    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:40:18.653723 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648989    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648991    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648994    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.648996    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649000    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649003    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649007    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649010    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649012    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649015    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649017    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649020    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649022    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649025    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649028    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649030    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649033    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649036    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649038    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649041    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:40:18.654222 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649043    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649114    2566 flags.go:64] FLAG: --address="0.0.0.0"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649147    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649153    2566 flags.go:64] FLAG: --anonymous-auth="true"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649157    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649162    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649165    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649169    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649173    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649176    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649179    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649182    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649186    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649189    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649192    2566 flags.go:64] FLAG: --cgroup-root=""
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649195    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649197    2566 flags.go:64] FLAG: --client-ca-file=""
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649200    2566 flags.go:64] FLAG: --cloud-provider="external"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649206    2566 flags.go:64] FLAG: --cluster-dns="[]"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649212    2566 flags.go:64] FLAG: --cluster-domain=""
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649215    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649218    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649222    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649225    2566 flags.go:64] FLAG: --container-log-max-files="5"
Sep 09 20:40:18.654727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649229    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649231    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649234    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649238    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649241    2566 flags.go:64] FLAG: --contention-profiling="false"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649245    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649248    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649251    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649254    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649258    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649261    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649264    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649267    2566 flags.go:64] FLAG: --enable-load-reader="false"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649270    2566 flags.go:64] FLAG: --enable-server="true"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649274    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649278    2566 flags.go:64] FLAG: --event-burst="100"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649281    2566 flags.go:64] FLAG: --event-qps="50"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649284    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649286    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649289    2566 flags.go:64] FLAG: --eviction-hard=""
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649293    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649295    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649298    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649301    2566 flags.go:64] FLAG: --eviction-soft=""
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649304    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 09 20:40:18.655290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649306    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649311    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649314    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649317    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649320    2566 flags.go:64] FLAG: --fail-swap-on="true"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649322    2566 flags.go:64] FLAG: --feature-gates=""
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649326    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649329    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649332    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649335    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649338    2566 flags.go:64] FLAG: --healthz-port="10248"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649341    2566 flags.go:64] FLAG: --help="false"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649345    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-137-187.ec2.internal"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649348    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649351    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649354    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649357    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649361    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649363    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649366    2566 flags.go:64] FLAG: --image-service-endpoint=""
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649369    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649372    2566 flags.go:64] FLAG: --kube-api-burst="100"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649375    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649378    2566 flags.go:64] FLAG: --kube-api-qps="50"
Sep 09 20:40:18.655928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649381    2566 flags.go:64] FLAG: --kube-reserved=""
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649384    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649386    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649389    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649392    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649395    2566 flags.go:64] FLAG: --lock-file=""
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649398    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649400    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649403    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649410    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649412    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649415    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649418    2566 flags.go:64] FLAG: --logging-format="text"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649421    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649424    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649427    2566 flags.go:64] FLAG: --manifest-url=""
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649430    2566 flags.go:64] FLAG: --manifest-url-header=""
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649438    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649441    2566 flags.go:64] FLAG: --max-open-files="1000000"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649446    2566 flags.go:64] FLAG: --max-pods="110"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649452    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649455    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649458    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649461    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649464    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 09 20:40:18.656503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649467    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649469    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649476    2566 flags.go:64] FLAG: --node-status-max-images="50"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649479    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649482    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649485    2566 flags.go:64] FLAG: --pod-cidr=""
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649489    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649494    2566 flags.go:64] FLAG: --pod-manifest-path=""
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649496    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649499    2566 flags.go:64] FLAG: --pods-per-core="0"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649502    2566 flags.go:64] FLAG: --port="10250"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649505    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649508    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0952f399b2bed6a39"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649511    2566 flags.go:64] FLAG: --qos-reserved=""
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649514    2566 flags.go:64] FLAG: --read-only-port="10255"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649517    2566 flags.go:64] FLAG: --register-node="true"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649520    2566 flags.go:64] FLAG: --register-with-taints=""
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649525    2566 flags.go:64] FLAG: --registry-burst="10"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649528    2566 flags.go:64] FLAG: --registry-qps="5"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649531    2566 flags.go:64] FLAG: --reserved-cpus=""
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649533    2566 flags.go:64] FLAG: --reserved-memory=""
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649537    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649540    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649543    2566 flags.go:64] FLAG: --rotate-certificates="false"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649546    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 09 20:40:18.657131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649548    2566 flags.go:64] FLAG: --runonce="false"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649551    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649555    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649558    2566 flags.go:64] FLAG: --seccomp-default="false"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649561    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649564    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649581    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649585    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649588    2566 flags.go:64] FLAG: --storage-driver-password="root"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649591    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649594    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649596    2566 flags.go:64] FLAG: --storage-driver-user="root"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649599    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649602    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649605    2566 flags.go:64] FLAG: --system-cgroups=""
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649608    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649614    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649616    2566 flags.go:64] FLAG: --tls-cert-file=""
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649619    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649623    2566 flags.go:64] FLAG: --tls-min-version=""
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649626    2566 flags.go:64] FLAG: --tls-private-key-file=""
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649628    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649631    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649634    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649638    2566 flags.go:64] FLAG: --v="2"
Sep 09 20:40:18.657790 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649642    2566 flags.go:64] FLAG: --version="false"
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649646    2566 flags.go:64] FLAG: --vmodule=""
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649650    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.649653    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649760    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649765    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649768    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649771    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649774    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649778    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649780    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649783    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649785    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649788    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649790    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649793    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649795    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649798    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649800    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649803    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:40:18.658391 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649806    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649809    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649812    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649814    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649817    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649820    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649822    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649825    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649827    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649829    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649832    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649836    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649838    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649841    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649844    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649846    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649849    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649851    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649854    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649857    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:40:18.658908 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649859    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649863    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649865    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649868    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649871    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649873    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649876    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649879    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649881    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649884    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649886    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649889    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649892    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649894    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649897    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649899    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649902    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649904    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649907    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649909    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:40:18.659402 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649912    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649914    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649917    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649920    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649923    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649927    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649931    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649935    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649938    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649942    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649945    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649947    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649950    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649954    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649957    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649960    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649962    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649965    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649967    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:40:18.659992 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649970    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649972    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649975    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649977    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649980    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649982    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649985    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649988    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649990    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649993    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649995    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.649998    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650000    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650003    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650005    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650007    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650011    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650014    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650016    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650019    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:40:18.660461 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650021    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650024    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650026    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.650029    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.650525    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.657803    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.657817    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.657836    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.657841    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657894    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657899    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657903    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657907    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657910    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:40:18.660952 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657913    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657916    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657919    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657922    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657925    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657928    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657931    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657935    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657938    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657941    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657943    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657946    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657948    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657951    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657954    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657957    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657959    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657962    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657964    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657967    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:40:18.661307 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657970    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657972    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657976    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657980    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657984    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657988    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657991    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657994    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.657997    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658001    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658004    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658006    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658009    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658012    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658015    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658017    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658020    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658022    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658025    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:40:18.661809 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658027    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658030    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658033    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658035    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658038    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658040    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658042    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658045    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658048    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658050    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658053    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658055    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658058    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658060    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658062    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658065    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658067    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658071    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658074    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658076    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:40:18.662274 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658079    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658081    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658084    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658087    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658089    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658092    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658095    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658097    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658100    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658102    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658105    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658107    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658110    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658112    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658115    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658118    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658120    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658123    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658127    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658130    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:40:18.662776 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658133    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658136    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658138    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658141    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658143    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658146    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658148    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658151    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658153    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658157    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658159    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658162    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658164    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658167    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658169    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:40:18.663262 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.658174    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658280    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658284    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658287    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658290    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658293    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658295    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658298    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658301    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658304    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658307    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658309    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658312    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658316    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658319    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658322    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658324    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658327    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658330    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 09 20:40:18.663688 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658333    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658336    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658338    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658341    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658343    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658346    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658348    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658351    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658354    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658357    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658360    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658362    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658364    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658367    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658369    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658372    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658375    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658377    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658379    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 09 20:40:18.664141 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658382    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658384    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658387    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658389    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658392    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658394    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658397    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658400    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658404    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658407    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658410    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658412    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658415    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658417    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658420    2566 feature_gate.go:328] unrecognized feature gate: Example
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658422    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658424    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658427    2566 feature_gate.go:328] unrecognized feature gate: Example2
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658429    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658432    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 09 20:40:18.664608 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658435    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658437    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658440    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658442    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658445    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658447    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658450    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658453    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658457    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658460    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658463    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658466    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658468    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658471    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658473    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658476    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658478    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658481    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658483    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658486    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 09 20:40:18.665077 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658489    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658491    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658493    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658496    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658498    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658501    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658503    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658506    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658508    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658510    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658513    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658515    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658518    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658521    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658524    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658526    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658528    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658531    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658533    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658536    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 09 20:40:18.665555 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658538    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 09 20:40:18.666042 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:18.658541    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 09 20:40:18.666042 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.658545    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 09 20:40:18.666042 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.658650    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 09 20:40:18.666042 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.662154    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 09 20:40:18.666042 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.663102    2566 server.go:1019] "Starting client certificate rotation"
Sep 09 20:40:18.666042 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.663200    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:40:18.666042 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.663238    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 09 20:40:18.667640 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.667610    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:40:18.671413 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.671390    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 09 20:40:18.679411 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.679391    2566 log.go:25] "Validated CRI v1 runtime API"
Sep 09 20:40:18.682507 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.682493    2566 log.go:25] "Validated CRI v1 image API"
Sep 09 20:40:18.684821 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.684803    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 09 20:40:18.688903 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.688884    2566 fs.go:135] Filesystem UUIDs: map[6520489c-c5a5-43df-b918-c5b24fe10aaf:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c8194e81-eb11-4d44-9b64-c238034fcfe2:/dev/nvme0n1p4]
Sep 09 20:40:18.688984 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.688901    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 09 20:40:18.693994 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.693799    2566 manager.go:216] Machine: {Timestamp:2026-09-09 20:40:18.692456717 +0000 UTC m=+0.370030894 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101971 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a66493a800a28141c0e73043a0192 SystemUUID:ec2a6649-3a80-0a28-141c-0e73043a0192 BootID:dc32bcd0-7aa9-4262-9410-46acdfb50f22 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:96:a5:47:69:f1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:96:a5:47:69:f1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:8e:53:af:1c:e3 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 09 20:40:18.693994 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.693985    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 09 20:40:18.694116 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.694056    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 09 20:40:18.697484 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.697461    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 09 20:40:18.697629 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.697485    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-187.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 09 20:40:18.697718 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.697667    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:40:18.698311 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.698301    2566 topology_manager.go:138] "Creating topology manager with none policy"
Sep 09 20:40:18.698346 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.698313    2566 container_manager_linux.go:309] "Creating device plugin manager"
Sep 09 20:40:18.698346 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.698326    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:40:18.698346 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.698339    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 09 20:40:18.700504 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.700494    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 09 20:40:18.705654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.705643    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:40:18.705766 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.705757    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 09 20:40:18.705798 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.705787    2566 kubelet.go:486] "Attempting to sync node with API server"
Sep 09 20:40:18.705798 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.705797    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 09 20:40:18.705917 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.705811    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 09 20:40:18.705917 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.705820    2566 kubelet.go:398] "Adding apiserver pod source"
Sep 09 20:40:18.705917 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.705830    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 09 20:40:18.706198 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.706186    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:40:18.706260 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.706209    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 09 20:40:18.709794 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.709778    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 09 20:40:18.712472 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.712459    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 09 20:40:18.712517 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.712488    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 09 20:40:18.712676 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.712663    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 09 20:40:18.712707 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.712687    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 09 20:40:18.712707 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.712694    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 09 20:40:18.712707 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.712699    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 09 20:40:18.712707 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.712707    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 09 20:40:18.712808 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.712712    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 09 20:40:18.713261 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.713251    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 09 20:40:18.713297 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.713261    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 09 20:40:18.713297 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.713268    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 09 20:40:18.713297 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.713273    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 09 20:40:18.713297 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.713282    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 09 20:40:18.713297 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.713289    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 09 20:40:18.714115 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.714106    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 09 20:40:18.714115 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.714115    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 09 20:40:18.717140 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.717125    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-187.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 09 20:40:18.717211 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.717169    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 09 20:40:18.717211 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.717175    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 09 20:40:18.717909 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.717897    2566 server.go:1268] "Started kubelet"
Sep 09 20:40:18.718095 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.718073    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 09 20:40:18.718865 ip-10-0-137-187 systemd[1]: Started Kubernetes Kubelet.
Sep 09 20:40:18.719937 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.719875    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 09 20:40:18.720014 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.719960    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 09 20:40:18.720647 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.720634    2566 server.go:310] "Adding debug handlers to kubelet server"
Sep 09 20:40:18.721259 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.721240    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 09 20:40:18.723971 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.722893    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-187.ec2.internal.18d3c1ab5bad4a94  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-187.ec2.internal,UID:ip-10-0-137-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-187.ec2.internal,},FirstTimestamp:2026-09-09 20:40:18.71787074 +0000 UTC m=+0.395444917,LastTimestamp:2026-09-09 20:40:18.71787074 +0000 UTC m=+0.395444917,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-187.ec2.internal,}"
Sep 09 20:40:18.725786 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.725766    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 09 20:40:18.725786 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.725781    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 09 20:40:18.725889 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.725783    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 09 20:40:18.725944 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.725930    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:18.726520 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.726503    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 09 20:40:18.726634 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.726551    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 09 20:40:18.726634 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.726565    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 09 20:40:18.726728 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.726641    2566 reconstruct.go:97] "Volume reconstruction finished"
Sep 09 20:40:18.726728 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.726651    2566 reconciler.go:29] "Reconciler: start to sync state"
Sep 09 20:40:18.726728 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.726719    2566 factory.go:55] Registering systemd factory
Sep 09 20:40:18.726859 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.726760    2566 factory.go:223] Registration of the systemd container factory successfully
Sep 09 20:40:18.727060 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.727043    2566 factory.go:153] Registering CRI-O factory
Sep 09 20:40:18.727147 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.727063    2566 factory.go:223] Registration of the crio container factory successfully
Sep 09 20:40:18.727147 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.727126    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 09 20:40:18.727147 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.727144    2566 factory.go:103] Registering Raw factory
Sep 09 20:40:18.727290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.727160    2566 manager.go:1196] Started watching for new ooms in manager
Sep 09 20:40:18.727545 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.727533    2566 manager.go:349] Starting recovery of all containers
Sep 09 20:40:18.737343 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.737314    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 09 20:40:18.737422 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.737325    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-187.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 09 20:40:18.740981 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.740966    2566 manager.go:354] Recovery completed
Sep 09 20:40:18.745396 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.745380    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:40:18.745894 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.745878    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnnw9"
Sep 09 20:40:18.747942 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.747929    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:40:18.747998 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.747954    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:40:18.747998 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.747965    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:40:18.748397 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.748383    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 09 20:40:18.748397 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.748394    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 09 20:40:18.748477 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.748410    2566 state_mem.go:36] "Initialized new in-memory state store"
Sep 09 20:40:18.748477 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.748422    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:40:18.749173 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.749153    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:40:18.749223 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.749177    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:40:18.749223 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.749190    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:40:18.749886 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.749815    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-187.ec2.internal.18d3c1ab5d78265b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-187.ec2.internal,UID:ip-10-0-137-187.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-137-187.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-137-187.ec2.internal,},FirstTimestamp:2026-09-09 20:40:18.747942491 +0000 UTC m=+0.425516668,LastTimestamp:2026-09-09 20:40:18.747942491 +0000 UTC m=+0.425516668,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-187.ec2.internal,}"
Sep 09 20:40:18.750633 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.750622    2566 policy_none.go:49] "None policy: Start"
Sep 09 20:40:18.750669 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.750636    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 09 20:40:18.750669 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.750646    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 09 20:40:18.752623 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.752608    2566 policy_none.go:47] "Start"
Sep 09 20:40:18.754107 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.754089    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-gnnw9"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.792383    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.793449    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.793492    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.793511    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.793551    2566 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.795741    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.796279    2566 manager.go:337] "Starting Device Plugin manager"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.796317    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.796330    2566 server.go:87] "Starting device plugin registration server"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.796709    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.796729    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.796908    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.796974    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.796983    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.797566    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 09 20:40:18.803759 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.797649    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:18.894052 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.893963    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal"]
Sep 09 20:40:18.894052 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.894059    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:40:18.895766 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.895747    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:40:18.895889 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.895786    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:40:18.895889 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.895799    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:40:18.896957 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.896944    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:40:18.897684 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.897665    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:40:18.897749 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.897698    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:40:18.897749 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.897713    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:40:18.897959 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.897947    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:40:18.898115 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.898100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:18.898169 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.898133    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:40:18.898209 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.898200    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-187.ec2.internal"
Sep 09 20:40:18.898625 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.898607    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:40:18.898758 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.898637    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:40:18.898758 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.898652    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:40:18.898995 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.898971    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:40:18.899093 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.898999    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:40:18.899093 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.899013    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:40:18.900741 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.900725    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:18.900839 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.900749    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 09 20:40:18.902276 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.902261    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientMemory"
Sep 09 20:40:18.902361 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.902287    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasNoDiskPressure"
Sep 09 20:40:18.902361 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.902296    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeHasSufficientPID"
Sep 09 20:40:18.906517 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:18.906503    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-187.ec2.internal"
Sep 09 20:40:18.906587 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.906522    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-187.ec2.internal\": node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:18.932241 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.932222    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:18.939194 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.939179    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-187.ec2.internal\" not found" node="ip-10-0-137-187.ec2.internal"
Sep 09 20:40:18.943667 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:18.943653    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-187.ec2.internal\" not found" node="ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.028396 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.028374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6cc7b2849d932e3b904e6c8fc3f867d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal\" (UID: \"6cc7b2849d932e3b904e6c8fc3f867d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.028483 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.028398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cc7b2849d932e3b904e6c8fc3f867d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal\" (UID: \"6cc7b2849d932e3b904e6c8fc3f867d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.028483 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.028415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a5dfa25977fd74cf93272977ce002f0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-187.ec2.internal\" (UID: \"9a5dfa25977fd74cf93272977ce002f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.032460 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:19.032443    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:19.128852 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.128826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6cc7b2849d932e3b904e6c8fc3f867d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal\" (UID: \"6cc7b2849d932e3b904e6c8fc3f867d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.128951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.128853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cc7b2849d932e3b904e6c8fc3f867d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal\" (UID: \"6cc7b2849d932e3b904e6c8fc3f867d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.128951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.128872    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a5dfa25977fd74cf93272977ce002f0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-187.ec2.internal\" (UID: \"9a5dfa25977fd74cf93272977ce002f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.128951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.128898    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/9a5dfa25977fd74cf93272977ce002f0-config\") pod \"kube-apiserver-proxy-ip-10-0-137-187.ec2.internal\" (UID: \"9a5dfa25977fd74cf93272977ce002f0\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.128951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.128917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6cc7b2849d932e3b904e6c8fc3f867d7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal\" (UID: \"6cc7b2849d932e3b904e6c8fc3f867d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.128951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.128917    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6cc7b2849d932e3b904e6c8fc3f867d7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal\" (UID: \"6cc7b2849d932e3b904e6c8fc3f867d7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.132937 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:19.132921    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:19.233805 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:19.233743    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:19.242898 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.242879    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.248244 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:19.248218    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:19.334772 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:19.334736    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:19.435213 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:19.435187    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:21.021477 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.021459    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-187.ec2.internal\" not found"
Sep 09 20:40:21.037318 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.037290    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:40:21.112152 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.112130    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:21.126442 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.126424    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:40:21.128532 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.128519    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal"
Sep 09 20:40:21.136645 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.136629    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 09 20:40:21.148777 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.148759    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 09 20:40:21.148890 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.148871    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:40:21.148890 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.148879    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 09 20:40:21.192057 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.192031    2566 apiserver.go:52] "Watching apiserver"
Sep 09 20:40:21.207085 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.207061    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 09 20:40:21.209006 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.208986    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-8rk4n","openshift-network-operator/iptables-alerter-d9phf","openshift-ovn-kubernetes/ovnkube-node-6n527","kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l","openshift-dns/node-resolver-qdsbk","openshift-image-registry/node-ca-g2d92","openshift-multus/multus-additional-cni-plugins-r2vzw","openshift-multus/multus-z5c8t","kube-system/global-pull-secret-syncer-zm7nt","kube-system/konnectivity-agent-55k2q","openshift-cluster-node-tuning-operator/tuned-5r4wn","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal","openshift-multus/network-metrics-daemon-qk9rl"]
Sep 09 20:40:21.211348 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.211326    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.211426 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.211385    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:21.211789 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.211776    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 09 20:40:21.213329 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.213316    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:21.215416 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.215395    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 09 20:40:21.215517 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.215432    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 09 20:40:21.215517 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.215454    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4mbpn\""
Sep 09 20:40:21.215614 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.215538    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.217434 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.217419    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 09 20:40:21.217787 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.217773    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:21.217862 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.217832    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 09 20:40:21.217918 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.217862    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:21.217981 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.217966    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 09 20:40:21.218098 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.218083    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 09 20:40:21.218139 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.218123    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 09 20:40:21.218177 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.218140    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 09 20:40:21.218177 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.218160    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-w54k8\""
Sep 09 20:40:21.218371 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.218356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fe41335-60ef-4123-99c6-0c504cf31555-kubelet-config\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.218413 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.218379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fe41335-60ef-4123-99c6-0c504cf31555-dbus\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.218413 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.218394    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.218500 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.218415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35cb7c42-1626-4c65-bde3-0f84790c896b-agent-certs\") pod \"konnectivity-agent-55k2q\" (UID: \"35cb7c42-1626-4c65-bde3-0f84790c896b\") " pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:21.218500 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.218437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35cb7c42-1626-4c65-bde3-0f84790c896b-konnectivity-ca\") pod \"konnectivity-agent-55k2q\" (UID: \"35cb7c42-1626-4c65-bde3-0f84790c896b\") " pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:21.219893 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.219878    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.221881 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.221867    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-28xmp\""
Sep 09 20:40:21.222050 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.222033    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 09 20:40:21.222152 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.222137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.222484 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.222469    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 09 20:40:21.222552 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.222499    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 09 20:40:21.224338 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.224319    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 09 20:40:21.224435 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.224418    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-b5ntm\""
Sep 09 20:40:21.224495 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.224425    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 09 20:40:21.224546 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.224520    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.225213 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.225190    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 09 20:40:21.226615 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.226596    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 09 20:40:21.227509 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.227486    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 09 20:40:21.227675 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.227657    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 09 20:40:21.227772 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.227728    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 09 20:40:21.227839 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.227811    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.228015 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.227991    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-7d425\""
Sep 09 20:40:21.228214 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.228198    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:40:21.229945 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.229923    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.230095 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.230074    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 09 20:40:21.230517 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.230498    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 09 20:40:21.230517 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.230509    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-cvldf\""
Sep 09 20:40:21.232255 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.232236    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 09 20:40:21.232347 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.232267    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 09 20:40:21.232411 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.232361    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:21.232492 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.232439    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:21.232553 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.232518    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jcxcx\""
Sep 09 20:40:21.233203 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.233189    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:40:21.234508 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.234493    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.236693 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.236677    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.238743 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.238730    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 09 20:40:21.238801 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.238731    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 09 20:40:21.238991 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.238975    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 09 20:40:21.239041 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.238985    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vng9c\""
Sep 09 20:40:21.239091 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.239045    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 09 20:40:21.239185 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.239172    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 09 20:40:21.239226 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.239185    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-7q7rm\""
Sep 09 20:40:21.243039 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.242998    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-08 20:35:20 +0000 UTC" deadline="2028-02-23 06:02:34.34714851 +0000 UTC"
Sep 09 20:40:21.243039 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.243036    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12753h22m13.104114477s"
Sep 09 20:40:21.251045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.251027    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-z2mt4"
Sep 09 20:40:21.259076 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.259060    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-z2mt4"
Sep 09 20:40:21.313522 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.313504    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 09 20:40:21.318755 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a17c66e6-0c67-4239-b851-28df6826582d-ovnkube-config\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.318856 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a17c66e6-0c67-4239-b851-28df6826582d-ovn-node-metrics-cert\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.318856 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-socket-dir-parent\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.318856 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-registration-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.318856 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b472e626-f237-4a9a-a307-fc33a70a570f-cnibin\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.318856 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-lib-modules\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.319106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318895    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2c9v\" (UniqueName: \"kubernetes.io/projected/485dcc7e-6aea-4422-959d-2562372a8766-kube-api-access-b2c9v\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.319106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318940    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fe41335-60ef-4123-99c6-0c504cf31555-dbus\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.319106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.319106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.318995    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f60f0088-211f-490c-8012-edec87817ca1-hosts-file\") pod \"node-resolver-qdsbk\" (UID: \"f60f0088-211f-490c-8012-edec87817ca1\") " pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.319106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-slash\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.319106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5xqb\" (UniqueName: \"kubernetes.io/projected/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-kube-api-access-c5xqb\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.319106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2fe41335-60ef-4123-99c6-0c504cf31555-dbus\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.319106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1b58eba-4403-419f-a9f7-375076dc2fe1-host-slash\") pod \"iptables-alerter-d9phf\" (UID: \"e1b58eba-4403-419f-a9f7-375076dc2fe1\") " pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319122    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485dcc7e-6aea-4422-959d-2562372a8766-tmp\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.319132    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319146    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-etc-openvswitch\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-run-openvswitch\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.319192    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret podName:2fe41335-60ef-4123-99c6-0c504cf31555 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:21.819172023 +0000 UTC m=+2.011009612 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret") pod "global-pull-secret-syncer-zm7nt" (UID: "2fe41335-60ef-4123-99c6-0c504cf31555") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319221    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-hostroot\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319238    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28f00347-ee67-4e91-ac2f-0dbe68892960-serviceca\") pod \"node-ca-g2d92\" (UID: \"28f00347-ee67-4e91-ac2f-0dbe68892960\") " pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-run-ovn\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319269    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-node-log\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319295    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b472e626-f237-4a9a-a307-fc33a70a570f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319332    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-modprobe-d\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319350    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-run\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/485dcc7e-6aea-4422-959d-2562372a8766-etc-tuned\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5nrd\" (UniqueName: \"kubernetes.io/projected/a17c66e6-0c67-4239-b851-28df6826582d-kube-api-access-b5nrd\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b472e626-f237-4a9a-a307-fc33a70a570f-system-cni-dir\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.319525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319468    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b472e626-f237-4a9a-a307-fc33a70a570f-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319491    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b472e626-f237-4a9a-a307-fc33a70a570f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319509    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg85k\" (UniqueName: \"kubernetes.io/projected/b472e626-f237-4a9a-a307-fc33a70a570f-kube-api-access-jg85k\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319523    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-conf-dir\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35cb7c42-1626-4c65-bde3-0f84790c896b-konnectivity-ca\") pod \"konnectivity-agent-55k2q\" (UID: \"35cb7c42-1626-4c65-bde3-0f84790c896b\") " pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-var-lib-openvswitch\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-log-socket\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319619    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-system-cni-dir\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319638    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-kubernetes\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319675    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-sysctl-conf\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319711    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn2sn\" (UniqueName: \"kubernetes.io/projected/cad3822b-6c99-478f-a17a-26ca43f5fd7e-kube-api-access-bn2sn\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-etc-kubernetes\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319790    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z579s\" (UniqueName: \"kubernetes.io/projected/f60f0088-211f-490c-8012-edec87817ca1-kube-api-access-z579s\") pod \"node-resolver-qdsbk\" (UID: \"f60f0088-211f-490c-8012-edec87817ca1\") " pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319815    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r75hh\" (UniqueName: \"kubernetes.io/projected/28f00347-ee67-4e91-ac2f-0dbe68892960-kube-api-access-r75hh\") pod \"node-ca-g2d92\" (UID: \"28f00347-ee67-4e91-ac2f-0dbe68892960\") " pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.320287 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319840    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-sysconfig\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319863    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-host\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-systemd-units\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a17c66e6-0c67-4239-b851-28df6826582d-ovnkube-script-lib\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.319980    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-var-lib-kubelet\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b472e626-f237-4a9a-a307-fc33a70a570f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/35cb7c42-1626-4c65-bde3-0f84790c896b-konnectivity-ca\") pod \"konnectivity-agent-55k2q\" (UID: \"35cb7c42-1626-4c65-bde3-0f84790c896b\") " pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-sysctl-d\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35cb7c42-1626-4c65-bde3-0f84790c896b-agent-certs\") pod \"konnectivity-agent-55k2q\" (UID: \"35cb7c42-1626-4c65-bde3-0f84790c896b\") " pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320142    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-cni-binary-copy\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-daemon-config\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320255    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1b58eba-4403-419f-a9f7-375076dc2fe1-iptables-alerter-script\") pod \"iptables-alerter-d9phf\" (UID: \"e1b58eba-4403-419f-a9f7-375076dc2fe1\") " pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-var-lib-kubelet\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320286    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-cni-bin\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a17c66e6-0c67-4239-b851-28df6826582d-env-overrides\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320357    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-run-multus-certs\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.321187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28f00347-ee67-4e91-ac2f-0dbe68892960-host\") pod \"node-ca-g2d92\" (UID: \"28f00347-ee67-4e91-ac2f-0dbe68892960\") " pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-systemd\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-kubelet\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-cni-netd\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320524    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-sys\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320542    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-cni-dir\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-var-lib-cni-multus\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320624    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dwh\" (UniqueName: \"kubernetes.io/projected/e1b58eba-4403-419f-a9f7-375076dc2fe1-kube-api-access-k4dwh\") pod \"iptables-alerter-d9phf\" (UID: \"e1b58eba-4403-419f-a9f7-375076dc2fe1\") " pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320644    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-sys-fs\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxvgg\" (UniqueName: \"kubernetes.io/projected/386d3859-f092-4019-a4e1-7dc0f8e12f7d-kube-api-access-gxvgg\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320671    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-run-netns\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-socket-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fe41335-60ef-4123-99c6-0c504cf31555-kubelet-config\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320740    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60f0088-211f-490c-8012-edec87817ca1-tmp-dir\") pod \"node-resolver-qdsbk\" (UID: \"f60f0088-211f-490c-8012-edec87817ca1\") " pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.322045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-cnibin\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.322833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2fe41335-60ef-4123-99c6-0c504cf31555-kubelet-config\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.322833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:21.322833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320820    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-run-netns\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.322833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320823    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 09 20:40:21.322833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-run-systemd\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.322833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-run-k8s-cni-cncf-io\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.322833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-var-lib-cni-bin\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.322833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.320926    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-device-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.324786 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.324595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/35cb7c42-1626-4c65-bde3-0f84790c896b-agent-certs\") pod \"konnectivity-agent-55k2q\" (UID: \"35cb7c42-1626-4c65-bde3-0f84790c896b\") " pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:21.421568 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421543    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/485dcc7e-6aea-4422-959d-2562372a8766-etc-tuned\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.421662 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5nrd\" (UniqueName: \"kubernetes.io/projected/a17c66e6-0c67-4239-b851-28df6826582d-kube-api-access-b5nrd\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.421662 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421594    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b472e626-f237-4a9a-a307-fc33a70a570f-system-cni-dir\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.421662 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b472e626-f237-4a9a-a307-fc33a70a570f-system-cni-dir\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.421768 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421703    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b472e626-f237-4a9a-a307-fc33a70a570f-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.421768 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b472e626-f237-4a9a-a307-fc33a70a570f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.421859 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg85k\" (UniqueName: \"kubernetes.io/projected/b472e626-f237-4a9a-a307-fc33a70a570f-kube-api-access-jg85k\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.421859 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-conf-dir\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.421955 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.421955 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-var-lib-openvswitch\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.421955 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421910    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-log-socket\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.421955 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-conf-dir\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.421955 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-system-cni-dir\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-kubernetes\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.421997    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-sysctl-conf\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422000    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-log-socket\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422023    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn2sn\" (UniqueName: \"kubernetes.io/projected/cad3822b-6c99-478f-a17a-26ca43f5fd7e-kube-api-access-bn2sn\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-var-lib-openvswitch\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422055    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-etc-kubernetes\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z579s\" (UniqueName: \"kubernetes.io/projected/f60f0088-211f-490c-8012-edec87817ca1-kube-api-access-z579s\") pod \"node-resolver-qdsbk\" (UID: \"f60f0088-211f-490c-8012-edec87817ca1\") " pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r75hh\" (UniqueName: \"kubernetes.io/projected/28f00347-ee67-4e91-ac2f-0dbe68892960-kube-api-access-r75hh\") pod \"node-ca-g2d92\" (UID: \"28f00347-ee67-4e91-ac2f-0dbe68892960\") " pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422168    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-system-cni-dir\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-sysconfig\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.422189 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422091    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-etc-kubernetes\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-host\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422236    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-sysctl-conf\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422243    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-systemd-units\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422323    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-sysconfig\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b472e626-f237-4a9a-a307-fc33a70a570f-cni-binary-copy\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-systemd-units\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422232    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-run-ovn-kubernetes\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-kubernetes\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b472e626-f237-4a9a-a307-fc33a70a570f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-host\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a17c66e6-0c67-4239-b851-28df6826582d-ovnkube-script-lib\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-var-lib-kubelet\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422501    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-var-lib-kubelet\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422569    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b472e626-f237-4a9a-a307-fc33a70a570f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422631    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-sysctl-d\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.422817 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422656    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-cni-binary-copy\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-daemon-config\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-etc-selinux\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1b58eba-4403-419f-a9f7-375076dc2fe1-iptables-alerter-script\") pod \"iptables-alerter-d9phf\" (UID: \"e1b58eba-4403-419f-a9f7-375076dc2fe1\") " pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-var-lib-kubelet\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422799    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-cni-bin\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422820    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a17c66e6-0c67-4239-b851-28df6826582d-env-overrides\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422840    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-run-multus-certs\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422863    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28f00347-ee67-4e91-ac2f-0dbe68892960-host\") pod \"node-ca-g2d92\" (UID: \"28f00347-ee67-4e91-ac2f-0dbe68892960\") " pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422886    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-systemd\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-kubelet\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-cni-netd\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422960    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-sys\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/a17c66e6-0c67-4239-b851-28df6826582d-ovnkube-script-lib\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.422779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-sysctl-d\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.423654 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-cni-dir\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423089    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-var-lib-cni-multus\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423096    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-run-multus-certs\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dwh\" (UniqueName: \"kubernetes.io/projected/e1b58eba-4403-419f-a9f7-375076dc2fe1-kube-api-access-k4dwh\") pod \"iptables-alerter-d9phf\" (UID: \"e1b58eba-4403-419f-a9f7-375076dc2fe1\") " pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-cni-netd\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-sys-fs\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-kubelet\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423198    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-var-lib-kubelet\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423199    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxvgg\" (UniqueName: \"kubernetes.io/projected/386d3859-f092-4019-a4e1-7dc0f8e12f7d-kube-api-access-gxvgg\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423240    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-cni-binary-copy\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.423266    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423274    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-run-netns\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423303    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-cni-dir\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-sys\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423343    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/28f00347-ee67-4e91-ac2f-0dbe68892960-host\") pod \"node-ca-g2d92\" (UID: \"28f00347-ee67-4e91-ac2f-0dbe68892960\") " pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.423339    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs podName:cad3822b-6c99-478f-a17a-26ca43f5fd7e nodeName:}" failed. No retries permitted until 2026-09-09 20:40:21.92330037 +0000 UTC m=+2.115137961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs") pod "network-metrics-daemon-qk9rl" (UID: "cad3822b-6c99-478f-a17a-26ca43f5fd7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423247    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-run-netns\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.424454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423397    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-socket-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-sys-fs\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423433    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60f0088-211f-490c-8012-edec87817ca1-tmp-dir\") pod \"node-resolver-qdsbk\" (UID: \"f60f0088-211f-490c-8012-edec87817ca1\") " pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-cnibin\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423509    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423538    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-run-netns\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423560    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-daemon-config\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-run-systemd\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423569    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-var-lib-cni-multus\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-run-k8s-cni-cncf-io\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-var-lib-cni-bin\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-cni-bin\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-device-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423674    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-cnibin\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a17c66e6-0c67-4239-b851-28df6826582d-ovnkube-config\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-device-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423092    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-systemd\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.425331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423730    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-run-k8s-cni-cncf-io\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a17c66e6-0c67-4239-b851-28df6826582d-ovn-node-metrics-cert\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-run-netns\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423790    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-socket-dir-parent\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423806    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-host-var-lib-cni-bin\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-registration-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423824    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-run-systemd\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423841    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b472e626-f237-4a9a-a307-fc33a70a570f-cnibin\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-lib-modules\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423880    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-registration-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2c9v\" (UniqueName: \"kubernetes.io/projected/485dcc7e-6aea-4422-959d-2562372a8766-kube-api-access-b2c9v\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/a17c66e6-0c67-4239-b851-28df6826582d-env-overrides\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423932    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-multus-socket-dir-parent\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/386d3859-f092-4019-a4e1-7dc0f8e12f7d-socket-dir\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423952    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f60f0088-211f-490c-8012-edec87817ca1-hosts-file\") pod \"node-resolver-qdsbk\" (UID: \"f60f0088-211f-490c-8012-edec87817ca1\") " pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b472e626-f237-4a9a-a307-fc33a70a570f-cnibin\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.423976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-slash\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.426216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5xqb\" (UniqueName: \"kubernetes.io/projected/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-kube-api-access-c5xqb\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1b58eba-4403-419f-a9f7-375076dc2fe1-host-slash\") pod \"iptables-alerter-d9phf\" (UID: \"e1b58eba-4403-419f-a9f7-375076dc2fe1\") " pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485dcc7e-6aea-4422-959d-2562372a8766-tmp\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424074    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-lib-modules\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424106    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f60f0088-211f-490c-8012-edec87817ca1-tmp-dir\") pod \"node-resolver-qdsbk\" (UID: \"f60f0088-211f-490c-8012-edec87817ca1\") " pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424119    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-etc-openvswitch\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424128    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-host-slash\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424144    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-run-openvswitch\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-hostroot\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424178    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/f60f0088-211f-490c-8012-edec87817ca1-hosts-file\") pod \"node-resolver-qdsbk\" (UID: \"f60f0088-211f-490c-8012-edec87817ca1\") " pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424226    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28f00347-ee67-4e91-ac2f-0dbe68892960-serviceca\") pod \"node-ca-g2d92\" (UID: \"28f00347-ee67-4e91-ac2f-0dbe68892960\") " pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e1b58eba-4403-419f-a9f7-375076dc2fe1-host-slash\") pod \"iptables-alerter-d9phf\" (UID: \"e1b58eba-4403-419f-a9f7-375076dc2fe1\") " pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424261    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/a17c66e6-0c67-4239-b851-28df6826582d-ovnkube-config\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424329    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-etc-openvswitch\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424335    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-run-openvswitch\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424341    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-hostroot\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-run-ovn\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-node-log\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b472e626-f237-4a9a-a307-fc33a70a570f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424540    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-modprobe-d\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-run\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-run\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424699    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-run-ovn\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/a17c66e6-0c67-4239-b851-28df6826582d-node-log\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424839    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e1b58eba-4403-419f-a9f7-375076dc2fe1-iptables-alerter-script\") pod \"iptables-alerter-d9phf\" (UID: \"e1b58eba-4403-419f-a9f7-375076dc2fe1\") " pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b472e626-f237-4a9a-a307-fc33a70a570f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b472e626-f237-4a9a-a307-fc33a70a570f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.424968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/485dcc7e-6aea-4422-959d-2562372a8766-etc-modprobe-d\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.425497    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/485dcc7e-6aea-4422-959d-2562372a8766-etc-tuned\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.426044    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/28f00347-ee67-4e91-ac2f-0dbe68892960-serviceca\") pod \"node-ca-g2d92\" (UID: \"28f00347-ee67-4e91-ac2f-0dbe68892960\") " pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.426587    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/485dcc7e-6aea-4422-959d-2562372a8766-tmp\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.427920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.427609    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/a17c66e6-0c67-4239-b851-28df6826582d-ovn-node-metrics-cert\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.430501 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.430469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn2sn\" (UniqueName: \"kubernetes.io/projected/cad3822b-6c99-478f-a17a-26ca43f5fd7e-kube-api-access-bn2sn\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:21.430612 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.430450    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc7b2849d932e3b904e6c8fc3f867d7.slice/crio-94a273abd6ad7b20be251bb135abe6c63069cb00f80eb564e6dc2ebaa0e16de1 WatchSource:0}: Error finding container 94a273abd6ad7b20be251bb135abe6c63069cb00f80eb564e6dc2ebaa0e16de1: Status 404 returned error can't find the container with id 94a273abd6ad7b20be251bb135abe6c63069cb00f80eb564e6dc2ebaa0e16de1
Sep 09 20:40:21.430704 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.430683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg85k\" (UniqueName: \"kubernetes.io/projected/b472e626-f237-4a9a-a307-fc33a70a570f-kube-api-access-jg85k\") pod \"multus-additional-cni-plugins-r2vzw\" (UID: \"b472e626-f237-4a9a-a307-fc33a70a570f\") " pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.430905 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.430856    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r75hh\" (UniqueName: \"kubernetes.io/projected/28f00347-ee67-4e91-ac2f-0dbe68892960-kube-api-access-r75hh\") pod \"node-ca-g2d92\" (UID: \"28f00347-ee67-4e91-ac2f-0dbe68892960\") " pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.431037 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.431012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5nrd\" (UniqueName: \"kubernetes.io/projected/a17c66e6-0c67-4239-b851-28df6826582d-kube-api-access-b5nrd\") pod \"ovnkube-node-6n527\" (UID: \"a17c66e6-0c67-4239-b851-28df6826582d\") " pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.431193 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.431179    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:40:21.431236 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.431196    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:40:21.431236 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.431205    2566 projected.go:196] Error preparing data for projected volume kube-api-access-9pdpl for pod openshift-network-diagnostics/network-check-target-8rk4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:21.431298 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.431250    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl podName:cac22caa-f302-4b8e-bb6e-5eef356214b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:21.931237321 +0000 UTC m=+2.123074899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9pdpl" (UniqueName: "kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl") pod "network-check-target-8rk4n" (UID: "cac22caa-f302-4b8e-bb6e-5eef356214b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:21.432442 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.432415    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dwh\" (UniqueName: \"kubernetes.io/projected/e1b58eba-4403-419f-a9f7-375076dc2fe1-kube-api-access-k4dwh\") pod \"iptables-alerter-d9phf\" (UID: \"e1b58eba-4403-419f-a9f7-375076dc2fe1\") " pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.432669 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.432652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z579s\" (UniqueName: \"kubernetes.io/projected/f60f0088-211f-490c-8012-edec87817ca1-kube-api-access-z579s\") pod \"node-resolver-qdsbk\" (UID: \"f60f0088-211f-490c-8012-edec87817ca1\") " pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.433793 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.433772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2c9v\" (UniqueName: \"kubernetes.io/projected/485dcc7e-6aea-4422-959d-2562372a8766-kube-api-access-b2c9v\") pod \"tuned-5r4wn\" (UID: \"485dcc7e-6aea-4422-959d-2562372a8766\") " pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.433958 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.433934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxvgg\" (UniqueName: \"kubernetes.io/projected/386d3859-f092-4019-a4e1-7dc0f8e12f7d-kube-api-access-gxvgg\") pod \"aws-ebs-csi-driver-node-hjd6l\" (UID: \"386d3859-f092-4019-a4e1-7dc0f8e12f7d\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.434004 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.433992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5xqb\" (UniqueName: \"kubernetes.io/projected/e3cd2d69-db95-4669-8a7a-0c0e5d124a2e-kube-api-access-c5xqb\") pod \"multus-z5c8t\" (UID: \"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e\") " pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.437475 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.435329    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:40:21.459953 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.459931    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5dfa25977fd74cf93272977ce002f0.slice/crio-f3ac403ef79f51266f0b8a5fbb09adcd4cd3a509c6db251954b5049847dd226d WatchSource:0}: Error finding container f3ac403ef79f51266f0b8a5fbb09adcd4cd3a509c6db251954b5049847dd226d: Status 404 returned error can't find the container with id f3ac403ef79f51266f0b8a5fbb09adcd4cd3a509c6db251954b5049847dd226d
Sep 09 20:40:21.543352 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.543280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:21.551289 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.551267    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cb7c42_1626_4c65_bde3_0f84790c896b.slice/crio-b52085c90eff61e5962470ca6c92cdaae7692b59be978b2b8c54acb8c0f42b35 WatchSource:0}: Error finding container b52085c90eff61e5962470ca6c92cdaae7692b59be978b2b8c54acb8c0f42b35: Status 404 returned error can't find the container with id b52085c90eff61e5962470ca6c92cdaae7692b59be978b2b8c54acb8c0f42b35
Sep 09 20:40:21.557666 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.557647    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:21.565028 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.565003    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17c66e6_0c67_4239_b851_28df6826582d.slice/crio-b24014b2d8e8e997b4d3e5dc7b4f1be02f537770c795310ef4c01396c88b2eca WatchSource:0}: Error finding container b24014b2d8e8e997b4d3e5dc7b4f1be02f537770c795310ef4c01396c88b2eca: Status 404 returned error can't find the container with id b24014b2d8e8e997b4d3e5dc7b4f1be02f537770c795310ef4c01396c88b2eca
Sep 09 20:40:21.574801 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.574783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l"
Sep 09 20:40:21.582380 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.582361    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod386d3859_f092_4019_a4e1_7dc0f8e12f7d.slice/crio-4ffb1b379b60655515d95988f12cf4fe8a5d050e6fd6942872cc19b54ada2aff WatchSource:0}: Error finding container 4ffb1b379b60655515d95988f12cf4fe8a5d050e6fd6942872cc19b54ada2aff: Status 404 returned error can't find the container with id 4ffb1b379b60655515d95988f12cf4fe8a5d050e6fd6942872cc19b54ada2aff
Sep 09 20:40:21.587266 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.587251    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qdsbk"
Sep 09 20:40:21.594289 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.594266    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60f0088_211f_490c_8012_edec87817ca1.slice/crio-4437cefeee1ca83362644f0280cc59157a099e2c89a77aca60d4e43b09470081 WatchSource:0}: Error finding container 4437cefeee1ca83362644f0280cc59157a099e2c89a77aca60d4e43b09470081: Status 404 returned error can't find the container with id 4437cefeee1ca83362644f0280cc59157a099e2c89a77aca60d4e43b09470081
Sep 09 20:40:21.599093 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.599079    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5c8t"
Sep 09 20:40:21.606634 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.606616    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3cd2d69_db95_4669_8a7a_0c0e5d124a2e.slice/crio-71fa4057fcd7d2c866ea062050f476fbdc99a3d6928477f3fe3b193af7b9e10b WatchSource:0}: Error finding container 71fa4057fcd7d2c866ea062050f476fbdc99a3d6928477f3fe3b193af7b9e10b: Status 404 returned error can't find the container with id 71fa4057fcd7d2c866ea062050f476fbdc99a3d6928477f3fe3b193af7b9e10b
Sep 09 20:40:21.622837 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.622817    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-r2vzw"
Sep 09 20:40:21.630170 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.630151    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb472e626_f237_4a9a_a307_fc33a70a570f.slice/crio-e69797ffe0227a5fab6fecabb25ff98e97cbdcb1339c024e5af0de3560c56c14 WatchSource:0}: Error finding container e69797ffe0227a5fab6fecabb25ff98e97cbdcb1339c024e5af0de3560c56c14: Status 404 returned error can't find the container with id e69797ffe0227a5fab6fecabb25ff98e97cbdcb1339c024e5af0de3560c56c14
Sep 09 20:40:21.646572 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.646543    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-d9phf"
Sep 09 20:40:21.654502 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.654484    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b58eba_4403_419f_a9f7_375076dc2fe1.slice/crio-c50e95e7270a6ba78f26278760dcfedb3d97f6ffe02b658b412933010e707f89 WatchSource:0}: Error finding container c50e95e7270a6ba78f26278760dcfedb3d97f6ffe02b658b412933010e707f89: Status 404 returned error can't find the container with id c50e95e7270a6ba78f26278760dcfedb3d97f6ffe02b658b412933010e707f89
Sep 09 20:40:21.657227 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.657210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-g2d92"
Sep 09 20:40:21.661861 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.661846    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-5r4wn"
Sep 09 20:40:21.664711 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.664695    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f00347_ee67_4e91_ac2f_0dbe68892960.slice/crio-bfe939e0cbc8b1a5f92d5b5ff778f5a4302dde2eaf133b25016a3065ed2fc422 WatchSource:0}: Error finding container bfe939e0cbc8b1a5f92d5b5ff778f5a4302dde2eaf133b25016a3065ed2fc422: Status 404 returned error can't find the container with id bfe939e0cbc8b1a5f92d5b5ff778f5a4302dde2eaf133b25016a3065ed2fc422
Sep 09 20:40:21.666695 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.666682    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:40:21.669612 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:21.669595    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod485dcc7e_6aea_4422_959d_2562372a8766.slice/crio-fd502306c6b899fbad4c8c677049de349e27c3721a69f735dac90d573b713749 WatchSource:0}: Error finding container fd502306c6b899fbad4c8c677049de349e27c3721a69f735dac90d573b713749: Status 404 returned error can't find the container with id fd502306c6b899fbad4c8c677049de349e27c3721a69f735dac90d573b713749
Sep 09 20:40:21.827259 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.827192    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:21.827393 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.827326    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:21.827393 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.827375    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret podName:2fe41335-60ef-4123-99c6-0c504cf31555 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:22.827362424 +0000 UTC m=+3.019200000 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret") pod "global-pull-secret-syncer-zm7nt" (UID: "2fe41335-60ef-4123-99c6-0c504cf31555") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:21.928127 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:21.928095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:21.928295 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.928223    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:21.928295 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:21.928289    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs podName:cad3822b-6c99-478f-a17a-26ca43f5fd7e nodeName:}" failed. No retries permitted until 2026-09-09 20:40:22.928270995 +0000 UTC m=+3.120108586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs") pod "network-metrics-daemon-qk9rl" (UID: "cad3822b-6c99-478f-a17a-26ca43f5fd7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:22.029490 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.029459    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:22.029654 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:22.029630    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:40:22.029720 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:22.029654    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:40:22.029720 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:22.029669    2566 projected.go:196] Error preparing data for projected volume kube-api-access-9pdpl for pod openshift-network-diagnostics/network-check-target-8rk4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:22.029819 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:22.029732    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl podName:cac22caa-f302-4b8e-bb6e-5eef356214b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:23.029715081 +0000 UTC m=+3.221552664 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pdpl" (UniqueName: "kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl") pod "network-check-target-8rk4n" (UID: "cac22caa-f302-4b8e-bb6e-5eef356214b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:22.146607 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.146529    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:40:22.262401 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.262269    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:35:21 +0000 UTC" deadline="2028-06-06 01:31:56.08855837 +0000 UTC"
Sep 09 20:40:22.262401 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.262385    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15244h51m33.826177924s"
Sep 09 20:40:22.282861 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.282218    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:22.282861 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:22.282356    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:22.288800 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.288649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g2d92" event={"ID":"28f00347-ee67-4e91-ac2f-0dbe68892960","Type":"ContainerStarted","Data":"bfe939e0cbc8b1a5f92d5b5ff778f5a4302dde2eaf133b25016a3065ed2fc422"}
Sep 09 20:40:22.291917 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.291665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2vzw" event={"ID":"b472e626-f237-4a9a-a307-fc33a70a570f","Type":"ContainerStarted","Data":"e69797ffe0227a5fab6fecabb25ff98e97cbdcb1339c024e5af0de3560c56c14"}
Sep 09 20:40:22.298919 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.298892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5c8t" event={"ID":"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e","Type":"ContainerStarted","Data":"71fa4057fcd7d2c866ea062050f476fbdc99a3d6928477f3fe3b193af7b9e10b"}
Sep 09 20:40:22.306114 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.306089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qdsbk" event={"ID":"f60f0088-211f-490c-8012-edec87817ca1","Type":"ContainerStarted","Data":"4437cefeee1ca83362644f0280cc59157a099e2c89a77aca60d4e43b09470081"}
Sep 09 20:40:22.315737 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.315712    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-55k2q" event={"ID":"35cb7c42-1626-4c65-bde3-0f84790c896b","Type":"ContainerStarted","Data":"b52085c90eff61e5962470ca6c92cdaae7692b59be978b2b8c54acb8c0f42b35"}
Sep 09 20:40:22.318567 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.318540    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal" event={"ID":"9a5dfa25977fd74cf93272977ce002f0","Type":"ContainerStarted","Data":"f3ac403ef79f51266f0b8a5fbb09adcd4cd3a509c6db251954b5049847dd226d"}
Sep 09 20:40:22.320420 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.320396    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal" event={"ID":"6cc7b2849d932e3b904e6c8fc3f867d7","Type":"ContainerStarted","Data":"94a273abd6ad7b20be251bb135abe6c63069cb00f80eb564e6dc2ebaa0e16de1"}
Sep 09 20:40:22.324791 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.324767    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5r4wn" event={"ID":"485dcc7e-6aea-4422-959d-2562372a8766","Type":"ContainerStarted","Data":"fd502306c6b899fbad4c8c677049de349e27c3721a69f735dac90d573b713749"}
Sep 09 20:40:22.327207 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.327182    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d9phf" event={"ID":"e1b58eba-4403-419f-a9f7-375076dc2fe1","Type":"ContainerStarted","Data":"c50e95e7270a6ba78f26278760dcfedb3d97f6ffe02b658b412933010e707f89"}
Sep 09 20:40:22.332882 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.332860    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l" event={"ID":"386d3859-f092-4019-a4e1-7dc0f8e12f7d","Type":"ContainerStarted","Data":"4ffb1b379b60655515d95988f12cf4fe8a5d050e6fd6942872cc19b54ada2aff"}
Sep 09 20:40:22.340301 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.340276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" event={"ID":"a17c66e6-0c67-4239-b851-28df6826582d","Type":"ContainerStarted","Data":"b24014b2d8e8e997b4d3e5dc7b4f1be02f537770c795310ef4c01396c88b2eca"}
Sep 09 20:40:22.835872 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.835835    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:22.836065 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:22.835981    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:22.836065 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:22.836052    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret podName:2fe41335-60ef-4123-99c6-0c504cf31555 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:24.836024817 +0000 UTC m=+5.027862400 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret") pod "global-pull-secret-syncer-zm7nt" (UID: "2fe41335-60ef-4123-99c6-0c504cf31555") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:22.936488 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:22.936388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:22.936642 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:22.936533    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:22.936642 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:22.936596    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs podName:cad3822b-6c99-478f-a17a-26ca43f5fd7e nodeName:}" failed. No retries permitted until 2026-09-09 20:40:24.936578563 +0000 UTC m=+5.128416156 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs") pod "network-metrics-daemon-qk9rl" (UID: "cad3822b-6c99-478f-a17a-26ca43f5fd7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:23.037662 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:23.037387    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:23.037662 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:23.037568    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:40:23.037662 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:23.037586    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:40:23.037662 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:23.037599    2566 projected.go:196] Error preparing data for projected volume kube-api-access-9pdpl for pod openshift-network-diagnostics/network-check-target-8rk4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:23.037662 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:23.037653    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl podName:cac22caa-f302-4b8e-bb6e-5eef356214b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:25.037636259 +0000 UTC m=+5.229473839 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pdpl" (UniqueName: "kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl") pod "network-check-target-8rk4n" (UID: "cac22caa-f302-4b8e-bb6e-5eef356214b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:23.263281 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:23.263191    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-08 20:35:21 +0000 UTC" deadline="2028-06-15 21:38:20.941407242 +0000 UTC"
Sep 09 20:40:23.263281 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:23.263249    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15480h57m57.678162688s"
Sep 09 20:40:23.280904 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:23.280284    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:23.280904 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:23.280422    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:23.280904 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:23.280747    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:23.280904 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:23.280853    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:23.307297 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:23.307088    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 09 20:40:24.280519 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:24.280012    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:24.280519 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:24.280145    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:24.851338 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:24.850814    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:24.851338 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:24.851016    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:24.851338 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:24.851077    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret podName:2fe41335-60ef-4123-99c6-0c504cf31555 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:28.851059105 +0000 UTC m=+9.042896685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret") pod "global-pull-secret-syncer-zm7nt" (UID: "2fe41335-60ef-4123-99c6-0c504cf31555") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:24.951837 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:24.951801    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:24.952086 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:24.952067    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:24.952173 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:24.952144    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs podName:cad3822b-6c99-478f-a17a-26ca43f5fd7e nodeName:}" failed. No retries permitted until 2026-09-09 20:40:28.952123973 +0000 UTC m=+9.143961551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs") pod "network-metrics-daemon-qk9rl" (UID: "cad3822b-6c99-478f-a17a-26ca43f5fd7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:25.053248 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:25.052582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:25.053248 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:25.052812    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:40:25.053248 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:25.052834    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:40:25.053248 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:25.052847    2566 projected.go:196] Error preparing data for projected volume kube-api-access-9pdpl for pod openshift-network-diagnostics/network-check-target-8rk4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:25.053248 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:25.052907    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl podName:cac22caa-f302-4b8e-bb6e-5eef356214b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:29.052889238 +0000 UTC m=+9.244726836 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pdpl" (UniqueName: "kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl") pod "network-check-target-8rk4n" (UID: "cac22caa-f302-4b8e-bb6e-5eef356214b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:25.279718 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:25.279684    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:25.279891 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:25.279822    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:25.279959 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:25.279914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:25.280056 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:25.280034    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:26.280484 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:26.280452    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:26.281037 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:26.280588    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:27.279483 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:27.279453    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:27.279662 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:27.279453    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:27.279662 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:27.279585    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:27.279662 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:27.279648    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:28.279546 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:28.279504    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:28.280005 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:28.279618    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:28.884786 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:28.884241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:28.884786 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:28.884413    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:28.884786 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:28.884470    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret podName:2fe41335-60ef-4123-99c6-0c504cf31555 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:36.884451035 +0000 UTC m=+17.076288612 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret") pod "global-pull-secret-syncer-zm7nt" (UID: "2fe41335-60ef-4123-99c6-0c504cf31555") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:28.984767 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:28.984734    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:28.984940 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:28.984881    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:28.985002 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:28.984950    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs podName:cad3822b-6c99-478f-a17a-26ca43f5fd7e nodeName:}" failed. No retries permitted until 2026-09-09 20:40:36.984933855 +0000 UTC m=+17.176771435 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs") pod "network-metrics-daemon-qk9rl" (UID: "cad3822b-6c99-478f-a17a-26ca43f5fd7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:29.086230 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:29.085564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:29.086230 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:29.085753    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:40:29.086230 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:29.085777    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:40:29.086230 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:29.085788    2566 projected.go:196] Error preparing data for projected volume kube-api-access-9pdpl for pod openshift-network-diagnostics/network-check-target-8rk4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:29.086230 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:29.085846    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl podName:cac22caa-f302-4b8e-bb6e-5eef356214b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:37.085827942 +0000 UTC m=+17.277665533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pdpl" (UniqueName: "kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl") pod "network-check-target-8rk4n" (UID: "cac22caa-f302-4b8e-bb6e-5eef356214b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:29.279643 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:29.279605    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:29.280188 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:29.279605    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:29.280188 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:29.279746    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:29.280188 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:29.279821    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:30.281376 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:30.280917    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:30.281376 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:30.281030    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:31.280266 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:31.280227    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:31.280444 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:31.280227    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:31.280444 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:31.280364    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:31.280547 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:31.280463    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:32.279618 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:32.279579    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:32.280041 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:32.279737    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:33.280151 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:33.280124    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:33.280570 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:33.280258    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:33.280570 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:33.280326    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:33.280570 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:33.280435    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:34.280072 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:34.280033    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:34.280260 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:34.280166    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:35.280225 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:35.280192    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:35.280392 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:35.280330    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:35.280392 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:35.280370    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:35.280721 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:35.280472    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:36.280587 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:36.280550    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:36.281079 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:36.280684    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:36.940984 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:36.940945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:36.941204 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:36.941067    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:36.941204 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:36.941123    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret podName:2fe41335-60ef-4123-99c6-0c504cf31555 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:52.941108878 +0000 UTC m=+33.132946455 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret") pod "global-pull-secret-syncer-zm7nt" (UID: "2fe41335-60ef-4123-99c6-0c504cf31555") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:37.042116 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:37.042079    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:37.042281 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:37.042192    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:37.042281 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:37.042242    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs podName:cad3822b-6c99-478f-a17a-26ca43f5fd7e nodeName:}" failed. No retries permitted until 2026-09-09 20:40:53.042228449 +0000 UTC m=+33.234066026 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs") pod "network-metrics-daemon-qk9rl" (UID: "cad3822b-6c99-478f-a17a-26ca43f5fd7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:37.143367 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:37.143333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:37.143534 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:37.143475    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:40:37.143534 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:37.143492    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:40:37.143534 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:37.143504    2566 projected.go:196] Error preparing data for projected volume kube-api-access-9pdpl for pod openshift-network-diagnostics/network-check-target-8rk4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:37.143708 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:37.143552    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl podName:cac22caa-f302-4b8e-bb6e-5eef356214b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:53.14353951 +0000 UTC m=+33.335377087 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pdpl" (UniqueName: "kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl") pod "network-check-target-8rk4n" (UID: "cac22caa-f302-4b8e-bb6e-5eef356214b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:37.280323 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:37.280287    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:37.280504 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:37.280430    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:37.280504 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:37.280491    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:37.280617 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:37.280603    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:38.280427 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:38.280391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:38.280605 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:38.280519    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:39.279609 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:39.279579    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:39.280004 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:39.279738    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:39.280004 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:39.279753    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:39.280004 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:39.279848    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:40.281059 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:40.281022    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:40.281499 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:40.281128    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:41.280604 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.280331    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:41.280697 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.280385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:41.280730 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:41.280688    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:41.280802 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:41.280780    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:41.395504 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.395472    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-5r4wn" event={"ID":"485dcc7e-6aea-4422-959d-2562372a8766","Type":"ContainerStarted","Data":"cca4e89ed1d9c7d6d4fc79ef557edbc06d6d6912dae1c8de7e2f095a8cd90926"}
Sep 09 20:40:41.397114 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.397090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l" event={"ID":"386d3859-f092-4019-a4e1-7dc0f8e12f7d","Type":"ContainerStarted","Data":"ebf067e231c08d7521c6533c8a43fa7c60a97a0be034bde2b0facab32b3dfe51"}
Sep 09 20:40:41.400364 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.400342    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" event={"ID":"a17c66e6-0c67-4239-b851-28df6826582d","Type":"ContainerStarted","Data":"4af76ed38920dc31f84d4bdc6ff8fc07e37724c7d8030d5e9aea46f6ef241d6b"}
Sep 09 20:40:41.400476 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.400406    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" event={"ID":"a17c66e6-0c67-4239-b851-28df6826582d","Type":"ContainerStarted","Data":"d777c454bc99cad55a0c92e38e02beaf129cd3b6e371f55a0f7649026eb02bff"}
Sep 09 20:40:41.400476 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.400436    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" event={"ID":"a17c66e6-0c67-4239-b851-28df6826582d","Type":"ContainerStarted","Data":"8973fce235f30eaa663ab2376e8592ea0ba368bb5443ad5dd75aeda65ec33814"}
Sep 09 20:40:41.400476 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.400470    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" event={"ID":"a17c66e6-0c67-4239-b851-28df6826582d","Type":"ContainerStarted","Data":"79eafbd472eb3c40069d39ea1609d05770b35c79186efca7b8ba744633e77d2b"}
Sep 09 20:40:41.400629 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.400512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" event={"ID":"a17c66e6-0c67-4239-b851-28df6826582d","Type":"ContainerStarted","Data":"b1d17caa5ba1ec4ed1e67e372d2c176116d527f407fb8f4900a5b710c9879ae9"}
Sep 09 20:40:41.400629 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.400550    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" event={"ID":"a17c66e6-0c67-4239-b851-28df6826582d","Type":"ContainerStarted","Data":"ab6e1ff5a534fcaf0c6cb2310577000874aaf89fce405450db397ac8617a9292"}
Sep 09 20:40:41.401754 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.401730    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-g2d92" event={"ID":"28f00347-ee67-4e91-ac2f-0dbe68892960","Type":"ContainerStarted","Data":"1c29d2a915129df8f11b3c380ba7a40effb5d92db8baf0290c9e9a010cca8b0a"}
Sep 09 20:40:41.403181 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.403160    2566 generic.go:356] "Generic (PLEG): container finished" podID="b472e626-f237-4a9a-a307-fc33a70a570f" containerID="ef4f3155c69c7886e6d8b7c1400958445bbb61622d1d3585abc38800105f127f" exitCode=0
Sep 09 20:40:41.403282 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.403223    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2vzw" event={"ID":"b472e626-f237-4a9a-a307-fc33a70a570f","Type":"ContainerDied","Data":"ef4f3155c69c7886e6d8b7c1400958445bbb61622d1d3585abc38800105f127f"}
Sep 09 20:40:41.404893 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.404867    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5c8t" event={"ID":"e3cd2d69-db95-4669-8a7a-0c0e5d124a2e","Type":"ContainerStarted","Data":"097164ae81adfd063e35e49e49302233b96b87cc6918503bb52324c06cccc716"}
Sep 09 20:40:41.406420 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.406394    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qdsbk" event={"ID":"f60f0088-211f-490c-8012-edec87817ca1","Type":"ContainerStarted","Data":"48157dfb18a66c926953a64788b810f28f0a004c30d722a7d957d2aac6fa1758"}
Sep 09 20:40:41.407921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.407901    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-55k2q" event={"ID":"35cb7c42-1626-4c65-bde3-0f84790c896b","Type":"ContainerStarted","Data":"75dfc6ff721c2e50d43e8310b4118a3ee6add80d4aea1aaf20eceb0055f92019"}
Sep 09 20:40:41.409342 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.409299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal" event={"ID":"9a5dfa25977fd74cf93272977ce002f0","Type":"ContainerStarted","Data":"da832d65478604cc26d6782e7ea9cb3ad31c1e285489617dcdf76da6f39b5451"}
Sep 09 20:40:41.410793 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.410774    2566 generic.go:356] "Generic (PLEG): container finished" podID="6cc7b2849d932e3b904e6c8fc3f867d7" containerID="16acc3052aea05fc7ade90c8e516797529cffbde47b4371dcc376593870af097" exitCode=0
Sep 09 20:40:41.410870 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.410801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal" event={"ID":"6cc7b2849d932e3b904e6c8fc3f867d7","Type":"ContainerDied","Data":"16acc3052aea05fc7ade90c8e516797529cffbde47b4371dcc376593870af097"}
Sep 09 20:40:41.476553 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.476512    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-5r4wn" podStartSLOduration=2.70714726 podStartE2EDuration="21.476497336s" totalImagesPullingTime="18.769350076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:41.437032259 +0000 UTC m=+21.628869858" watchObservedRunningTime="2026-09-09 20:40:41.476497336 +0000 UTC m=+21.668334988"
Sep 09 20:40:41.499833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.499795    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-55k2q" podStartSLOduration=2.629663046 podStartE2EDuration="21.499780154s" totalImagesPullingTime="18.870117108s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:41.49948874 +0000 UTC m=+21.691326339" watchObservedRunningTime="2026-09-09 20:40:41.499780154 +0000 UTC m=+21.691617736"
Sep 09 20:40:41.547598 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.547550    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z5c8t" podStartSLOduration=2.681443368 podStartE2EDuration="21.547533852s" totalImagesPullingTime="18.866090484s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:41.528441718 +0000 UTC m=+21.720279317" watchObservedRunningTime="2026-09-09 20:40:41.547533852 +0000 UTC m=+21.739371450"
Sep 09 20:40:41.548148 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.548125    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-g2d92" podStartSLOduration=2.777076283 podStartE2EDuration="21.548117849s" totalImagesPullingTime="18.771041566s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:41.547497665 +0000 UTC m=+21.739335263" watchObservedRunningTime="2026-09-09 20:40:41.548117849 +0000 UTC m=+21.739955448"
Sep 09 20:40:41.599347 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.599248    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-187.ec2.internal" podStartSLOduration=20.599234457 podStartE2EDuration="20.599234457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:41.580523946 +0000 UTC m=+21.772361564" watchObservedRunningTime="2026-09-09 20:40:41.599234457 +0000 UTC m=+21.791072058"
Sep 09 20:40:41.599478 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:41.599364    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qdsbk" podStartSLOduration=2.771966112 podStartE2EDuration="21.599358182s" totalImagesPullingTime="18.82739207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:41.598934307 +0000 UTC m=+21.790771906" watchObservedRunningTime="2026-09-09 20:40:41.599358182 +0000 UTC m=+21.791195781"
Sep 09 20:40:42.280166 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:42.279959    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:42.280373 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:42.280273    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:42.414408 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:42.414361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal" event={"ID":"6cc7b2849d932e3b904e6c8fc3f867d7","Type":"ContainerStarted","Data":"6c4a2813514f3d2d144edb74a0d084c38ca1f370e0d28ee1f0596b5081d334b4"}
Sep 09 20:40:42.415823 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:42.415782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-d9phf" event={"ID":"e1b58eba-4403-419f-a9f7-375076dc2fe1","Type":"ContainerStarted","Data":"b77b30404fcd4de3d7f2cf1f0e16733cb5bf6f30e85954200f4ee32031cf65a6"}
Sep 09 20:40:42.430300 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:42.430260    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-187.ec2.internal" podStartSLOduration=21.430248912 podStartE2EDuration="21.430248912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:42.429455279 +0000 UTC m=+22.621292878" watchObservedRunningTime="2026-09-09 20:40:42.430248912 +0000 UTC m=+22.622086510"
Sep 09 20:40:42.445714 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:42.445671    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-d9phf" podStartSLOduration=3.662810119 podStartE2EDuration="22.445655675s" totalImagesPullingTime="18.782845556s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:42.445206573 +0000 UTC m=+22.637044171" watchObservedRunningTime="2026-09-09 20:40:42.445655675 +0000 UTC m=+22.637493278"
Sep 09 20:40:42.597943 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:42.597914    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 09 20:40:43.279449 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:43.279420    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:43.279449 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:43.279434    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:43.279678 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:43.279555    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:43.279678 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:43.279601    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:43.295837 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:43.295741    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-09T20:40:42.597938827Z","UUID":"325005a3-9bad-4a20-86b5-225197fdc165","Handler":null,"Name":"","Endpoint":""}
Sep 09 20:40:43.297836 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:43.297798    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 09 20:40:43.297951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:43.297848    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 09 20:40:43.420860 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:43.420821    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l" event={"ID":"386d3859-f092-4019-a4e1-7dc0f8e12f7d","Type":"ContainerStarted","Data":"4634d396c071243a3a586a85faaf1c84dbb316c159b3bc78474c435675010622"}
Sep 09 20:40:44.279485 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:44.279459    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:44.279650 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:44.279586    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:44.425439 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:44.425352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l" event={"ID":"386d3859-f092-4019-a4e1-7dc0f8e12f7d","Type":"ContainerStarted","Data":"322c2ebbff2fedae47f8a346a69fd83ea61b46047046880b4ac184f9b6fb4ca9"}
Sep 09 20:40:44.429194 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:44.429160    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" event={"ID":"a17c66e6-0c67-4239-b851-28df6826582d","Type":"ContainerStarted","Data":"2660eabac51bcb8804edc1a4d6e72d1f78e1ee1a5851b5a775f77a949f32652d"}
Sep 09 20:40:44.444653 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:44.444603    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hjd6l" podStartSLOduration=2.935121815 podStartE2EDuration="24.444584995s" totalImagesPullingTime="21.50946318s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:44.444142873 +0000 UTC m=+24.635980498" watchObservedRunningTime="2026-09-09 20:40:44.444584995 +0000 UTC m=+24.636422596"
Sep 09 20:40:45.279941 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:45.279914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:45.280136 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:45.279914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:45.280136 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:45.280056    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:45.280136 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:45.280093    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:46.280423 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:46.280395    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:46.280774 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:46.280493    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:46.364513 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:46.364484    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:46.365019 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:46.365000    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:46.432948 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:46.432928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:46.433399 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:46.433384    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-55k2q"
Sep 09 20:40:47.279916 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:47.279750    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:47.280045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:47.279807    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:47.280045 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:47.279988    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:47.280107 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:47.280049    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:47.436738 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:47.436710    2566 generic.go:356] "Generic (PLEG): container finished" podID="b472e626-f237-4a9a-a307-fc33a70a570f" containerID="6815c464baf1d7f5d67af42b5d73f523cab8bd1b61a0b003b29039edfaf0fbd9" exitCode=0
Sep 09 20:40:47.437279 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:47.436774    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2vzw" event={"ID":"b472e626-f237-4a9a-a307-fc33a70a570f","Type":"ContainerDied","Data":"6815c464baf1d7f5d67af42b5d73f523cab8bd1b61a0b003b29039edfaf0fbd9"}
Sep 09 20:40:47.440537 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:47.440518    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" event={"ID":"a17c66e6-0c67-4239-b851-28df6826582d","Type":"ContainerStarted","Data":"b9538717804d49d252b1b81b9aeb1bb8129af81ea075ff00c477d6f00b158950"}
Sep 09 20:40:47.482905 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:47.482864    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6n527" podStartSLOduration=8.290088668 podStartE2EDuration="27.482852726s" totalImagesPullingTime="19.192764058s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:47.482564044 +0000 UTC m=+27.674401643" watchObservedRunningTime="2026-09-09 20:40:47.482852726 +0000 UTC m=+27.674690326"
Sep 09 20:40:48.279766 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.279737    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:48.279980 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:48.279839    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:48.345114 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.345079    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zm7nt"]
Sep 09 20:40:48.345276 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.345210    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:48.345372 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:48.345340    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:48.348272 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.348222    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8rk4n"]
Sep 09 20:40:48.348964 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.348944    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qk9rl"]
Sep 09 20:40:48.349076 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.349062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:48.349208 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:48.349169    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:48.442824 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.442797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:48.443386 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.443222    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:48.443386 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.443292    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:48.443386 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.443359    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:48.443386 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:48.443350    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:48.457625 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.457604    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:48.457753 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:48.457731    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:40:49.446437 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:49.446408    2566 generic.go:356] "Generic (PLEG): container finished" podID="b472e626-f237-4a9a-a307-fc33a70a570f" containerID="ee10baa100d5802ea29c0ee90b8b19e71303d344a1efc43463256703bd1adf6c" exitCode=0
Sep 09 20:40:49.446901 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:49.446482    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2vzw" event={"ID":"b472e626-f237-4a9a-a307-fc33a70a570f","Type":"ContainerDied","Data":"ee10baa100d5802ea29c0ee90b8b19e71303d344a1efc43463256703bd1adf6c"}
Sep 09 20:40:50.281255 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:50.281076    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:50.281453 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:50.281111    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:50.281453 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:50.281342    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:50.281453 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:50.281134    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:50.281596 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:50.281402    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:50.281596 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:50.281503    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:51.452578 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:51.452543    2566 generic.go:356] "Generic (PLEG): container finished" podID="b472e626-f237-4a9a-a307-fc33a70a570f" containerID="3868c822b0f3b0a5da32930be10c9006119fefd1e2194f6530e8e59e0f59cce7" exitCode=0
Sep 09 20:40:51.452970 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:51.452596    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2vzw" event={"ID":"b472e626-f237-4a9a-a307-fc33a70a570f","Type":"ContainerDied","Data":"3868c822b0f3b0a5da32930be10c9006119fefd1e2194f6530e8e59e0f59cce7"}
Sep 09 20:40:52.280133 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:52.280103    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:52.280300 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:52.280103    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:52.280300 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:52.280242    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:52.280300 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:52.280288    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:52.280476 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:52.280433    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:52.280536 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:52.280524    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:52.956892 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:52.956850    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:52.957385 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:52.956996    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:52.957385 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:52.957075    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret podName:2fe41335-60ef-4123-99c6-0c504cf31555 nodeName:}" failed. No retries permitted until 2026-09-09 20:41:24.957058948 +0000 UTC m=+65.148896527 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret") pod "global-pull-secret-syncer-zm7nt" (UID: "2fe41335-60ef-4123-99c6-0c504cf31555") : object "kube-system"/"original-pull-secret" not registered
Sep 09 20:40:53.057821 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:53.057791    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:53.057987 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:53.057940    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:53.058060 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:53.058010    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs podName:cad3822b-6c99-478f-a17a-26ca43f5fd7e nodeName:}" failed. No retries permitted until 2026-09-09 20:41:25.057988762 +0000 UTC m=+65.249826340 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs") pod "network-metrics-daemon-qk9rl" (UID: "cad3822b-6c99-478f-a17a-26ca43f5fd7e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 09 20:40:53.158588 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:53.158556    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:53.158826 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:53.158766    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 09 20:40:53.158915 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:53.158854    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 09 20:40:53.158915 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:53.158870    2566 projected.go:196] Error preparing data for projected volume kube-api-access-9pdpl for pod openshift-network-diagnostics/network-check-target-8rk4n: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:53.159025 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:53.158951    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl podName:cac22caa-f302-4b8e-bb6e-5eef356214b2 nodeName:}" failed. No retries permitted until 2026-09-09 20:41:25.158913351 +0000 UTC m=+65.350750944 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-9pdpl" (UniqueName: "kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl") pod "network-check-target-8rk4n" (UID: "cac22caa-f302-4b8e-bb6e-5eef356214b2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 09 20:40:54.280150 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.280116    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:54.280892 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.280156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:54.280892 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.280235    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:54.280892 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.280244    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zm7nt" podUID="2fe41335-60ef-4123-99c6-0c504cf31555"
Sep 09 20:40:54.280892 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.280382    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8rk4n" podUID="cac22caa-f302-4b8e-bb6e-5eef356214b2"
Sep 09 20:40:54.280892 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.280462    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qk9rl" podUID="cad3822b-6c99-478f-a17a-26ca43f5fd7e"
Sep 09 20:40:54.598107 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.598041    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-187.ec2.internal" event="NodeReady"
Sep 09 20:40:54.598247 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.598159    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 09 20:40:54.639352 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.639323    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-cb854fdc9-m6sx9"]
Sep 09 20:40:54.643953 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.643932    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"]
Sep 09 20:40:54.644111 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.644092    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.646871 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.646851    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"]
Sep 09 20:40:54.647003 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.646975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:54.647003 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.646982    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 09 20:40:54.647218 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.647202    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tj77j\""
Sep 09 20:40:54.650154 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.650136    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"]
Sep 09 20:40:54.650296 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.650279    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.650385 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.650335    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"work-manager-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" type="*v1.Secret"
Sep 09 20:40:54.650448 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.650427    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 09 20:40:54.650448 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.650432    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 09 20:40:54.650578 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.650552    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Sep 09 20:40:54.650639 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.650582    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Sep 09 20:40:54.650733 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.650675    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret"
Sep 09 20:40:54.653704 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.653681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"
Sep 09 20:40:54.654129 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.654106    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 09 20:40:54.655861 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.655837    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 09 20:40:54.656008 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.655987    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 09 20:40:54.657199 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.657178    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 09 20:40:54.658026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.658003    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"]
Sep 09 20:40:54.658631 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.658615    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 09 20:40:54.658869 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.658837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb854fdc9-m6sx9"]
Sep 09 20:40:54.658991 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.658972    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-2brw4\""
Sep 09 20:40:54.660549 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.660529    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 09 20:40:54.672711 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.672687    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rxdcl"]
Sep 09 20:40:54.675938 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.675921    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"]
Sep 09 20:40:54.675938 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.675946    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"]
Sep 09 20:40:54.676095 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.676026    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rxdcl"
Sep 09 20:40:54.679472 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.679454    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 09 20:40:54.679746 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.679726    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 09 20:40:54.680177 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.680163    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-87t46\""
Sep 09 20:40:54.680510 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.680490    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 09 20:40:54.684179 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.684158    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-jhwgx"]
Sep 09 20:40:54.687127 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.687107    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.689388 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.689368    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rxdcl"]
Sep 09 20:40:54.689599 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.689550    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 09 20:40:54.689754 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.689719    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 09 20:40:54.689754 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.689719    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vs5mb\""
Sep 09 20:40:54.700458 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.700410    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhwgx"]
Sep 09 20:40:54.770605 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b863620-9ac7-4433-bba7-ed209457f8cf-config-volume\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.770730 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adb51729-08fc-4692-8ec2-be71226c33ec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf\" (UID: \"adb51729-08fc-4692-8ec2-be71226c33ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"
Sep 09 20:40:54.770730 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770647    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78303679-3153-4d46-91c0-dd93139fa25d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.770730 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770700    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dplpx\" (UniqueName: \"kubernetes.io/projected/adb51729-08fc-4692-8ec2-be71226c33ec-kube-api-access-dplpx\") pod \"managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf\" (UID: \"adb51729-08fc-4692-8ec2-be71226c33ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"
Sep 09 20:40:54.770876 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.770876 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770803    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/715ed8a6-4d2e-4169-8dc5-e72fe72850cd-tmp\") pod \"klusterlet-addon-workmgr-8bb9cfb9f-h4wx9\" (UID: \"715ed8a6-4d2e-4169-8dc5-e72fe72850cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:54.770876 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770823    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6rm7\" (UniqueName: \"kubernetes.io/projected/e562a150-9463-4b3d-a8be-a04e8703a450-kube-api-access-q6rm7\") pod \"ingress-canary-rxdcl\" (UID: \"e562a150-9463-4b3d-a8be-a04e8703a450\") " pod="openshift-ingress-canary/ingress-canary-rxdcl"
Sep 09 20:40:54.770876 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8596x\" (UniqueName: \"kubernetes.io/projected/78303679-3153-4d46-91c0-dd93139fa25d-kube-api-access-8596x\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.770876 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770869    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs5ch\" (UniqueName: \"kubernetes.io/projected/5b863620-9ac7-4433-bba7-ed209457f8cf-kube-api-access-qs5ch\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.771103 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770891    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-image-registry-private-configuration\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.771103 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770918    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-ca-trust-extracted\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.771103 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770944    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.771103 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.770966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cplnn\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-kube-api-access-cplnn\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.771103 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771013    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/715ed8a6-4d2e-4169-8dc5-e72fe72850cd-klusterlet-config\") pod \"klusterlet-addon-workmgr-8bb9cfb9f-h4wx9\" (UID: \"715ed8a6-4d2e-4169-8dc5-e72fe72850cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:54.771103 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771030    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.771103 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771045    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-certificates\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.771103 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771083    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.771103 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771097    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-bound-sa-token\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.771514 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771160    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-installation-pull-secrets\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.771514 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmb6\" (UniqueName: \"kubernetes.io/projected/715ed8a6-4d2e-4169-8dc5-e72fe72850cd-kube-api-access-fdmb6\") pod \"klusterlet-addon-workmgr-8bb9cfb9f-h4wx9\" (UID: \"715ed8a6-4d2e-4169-8dc5-e72fe72850cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:54.771514 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e562a150-9463-4b3d-a8be-a04e8703a450-cert\") pod \"ingress-canary-rxdcl\" (UID: \"e562a150-9463-4b3d-a8be-a04e8703a450\") " pod="openshift-ingress-canary/ingress-canary-rxdcl"
Sep 09 20:40:54.771514 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771273    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-ca\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.771514 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771291    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-hub\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.771514 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771319    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b863620-9ac7-4433-bba7-ed209457f8cf-tmp-dir\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.771514 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.771343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-trusted-ca\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.871691 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.871624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8596x\" (UniqueName: \"kubernetes.io/projected/78303679-3153-4d46-91c0-dd93139fa25d-kube-api-access-8596x\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.871691 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.871652    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs5ch\" (UniqueName: \"kubernetes.io/projected/5b863620-9ac7-4433-bba7-ed209457f8cf-kube-api-access-qs5ch\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.871691 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.871671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-image-registry-private-configuration\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.871691 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.871695    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-ca-trust-extracted\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.872006 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.871729    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.872006 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.871754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cplnn\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-kube-api-access-cplnn\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.872006 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.871783    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/715ed8a6-4d2e-4169-8dc5-e72fe72850cd-klusterlet-config\") pod \"klusterlet-addon-workmgr-8bb9cfb9f-h4wx9\" (UID: \"715ed8a6-4d2e-4169-8dc5-e72fe72850cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:54.872006 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.871806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.872006 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.871831    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:40:54.872006 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.871850    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb854fdc9-m6sx9: secret "image-registry-tls" not found
Sep 09 20:40:54.872006 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.871900    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls podName:a8752ff6-9b1b-4b50-8ece-1f045b1393f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:55.371881457 +0000 UTC m=+35.563719051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls") pod "image-registry-cb854fdc9-m6sx9" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4") : secret "image-registry-tls" not found
Sep 09 20:40:54.872006 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.871832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-certificates\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-bound-sa-token\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-installation-pull-secrets\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-ca-trust-extracted\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872147    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmb6\" (UniqueName: \"kubernetes.io/projected/715ed8a6-4d2e-4169-8dc5-e72fe72850cd-kube-api-access-fdmb6\") pod \"klusterlet-addon-workmgr-8bb9cfb9f-h4wx9\" (UID: \"715ed8a6-4d2e-4169-8dc5-e72fe72850cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872170    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e562a150-9463-4b3d-a8be-a04e8703a450-cert\") pod \"ingress-canary-rxdcl\" (UID: \"e562a150-9463-4b3d-a8be-a04e8703a450\") " pod="openshift-ingress-canary/ingress-canary-rxdcl"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-ca\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872217    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-hub\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b863620-9ac7-4433-bba7-ed209457f8cf-tmp-dir\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872262    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-trusted-ca\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b863620-9ac7-4433-bba7-ed209457f8cf-config-volume\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872342    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adb51729-08fc-4692-8ec2-be71226c33ec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf\" (UID: \"adb51729-08fc-4692-8ec2-be71226c33ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78303679-3153-4d46-91c0-dd93139fa25d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872413    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dplpx\" (UniqueName: \"kubernetes.io/projected/adb51729-08fc-4692-8ec2-be71226c33ec-kube-api-access-dplpx\") pod \"managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf\" (UID: \"adb51729-08fc-4692-8ec2-be71226c33ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872422    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-certificates\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.872496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872439    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.873186 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.872514    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:40:54.873186 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872522    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/715ed8a6-4d2e-4169-8dc5-e72fe72850cd-tmp\") pod \"klusterlet-addon-workmgr-8bb9cfb9f-h4wx9\" (UID: \"715ed8a6-4d2e-4169-8dc5-e72fe72850cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:54.873186 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:54.872565    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls podName:5b863620-9ac7-4433-bba7-ed209457f8cf nodeName:}" failed. No retries permitted until 2026-09-09 20:40:55.372544352 +0000 UTC m=+35.564381934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls") pod "dns-default-jhwgx" (UID: "5b863620-9ac7-4433-bba7-ed209457f8cf") : secret "dns-default-metrics-tls" not found
Sep 09 20:40:54.873186 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872590    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6rm7\" (UniqueName: \"kubernetes.io/projected/e562a150-9463-4b3d-a8be-a04e8703a450-kube-api-access-q6rm7\") pod \"ingress-canary-rxdcl\" (UID: \"e562a150-9463-4b3d-a8be-a04e8703a450\") " pod="openshift-ingress-canary/ingress-canary-rxdcl"
Sep 09 20:40:54.873186 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.872944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5b863620-9ac7-4433-bba7-ed209457f8cf-config-volume\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.873413 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.873264    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/715ed8a6-4d2e-4169-8dc5-e72fe72850cd-tmp\") pod \"klusterlet-addon-workmgr-8bb9cfb9f-h4wx9\" (UID: \"715ed8a6-4d2e-4169-8dc5-e72fe72850cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:54.873501 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.873478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b863620-9ac7-4433-bba7-ed209457f8cf-tmp-dir\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.875041 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.874875    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-trusted-ca\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.877504 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.877479    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-hub\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.877931 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.877904    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-installation-pull-secrets\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.878037 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.878002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-ca\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.878097 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.878041    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e562a150-9463-4b3d-a8be-a04e8703a450-cert\") pod \"ingress-canary-rxdcl\" (UID: \"e562a150-9463-4b3d-a8be-a04e8703a450\") " pod="openshift-ingress-canary/ingress-canary-rxdcl"
Sep 09 20:40:54.878097 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.878083    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.878203 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.878082    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/adb51729-08fc-4692-8ec2-be71226c33ec-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf\" (UID: \"adb51729-08fc-4692-8ec2-be71226c33ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"
Sep 09 20:40:54.878325 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.878286    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/78303679-3153-4d46-91c0-dd93139fa25d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:54.878477 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.878451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-image-registry-private-configuration\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.884543 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.884488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-bound-sa-token\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.884928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.884905    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs5ch\" (UniqueName: \"kubernetes.io/projected/5b863620-9ac7-4433-bba7-ed209457f8cf-kube-api-access-qs5ch\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:54.885041 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.884915    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cplnn\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-kube-api-access-cplnn\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:54.885384 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.885367    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6rm7\" (UniqueName: \"kubernetes.io/projected/e562a150-9463-4b3d-a8be-a04e8703a450-kube-api-access-q6rm7\") pod \"ingress-canary-rxdcl\" (UID: \"e562a150-9463-4b3d-a8be-a04e8703a450\") " pod="openshift-ingress-canary/ingress-canary-rxdcl"
Sep 09 20:40:54.903579 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.903546    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:54.990259 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:54.990214    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:55.012762 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.012739    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rxdcl"
Sep 09 20:40:55.376426 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.376385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:55.376891 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.376489    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:55.376891 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:55.376580    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:40:55.376891 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:55.376612    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb854fdc9-m6sx9: secret "image-registry-tls" not found
Sep 09 20:40:55.376891 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:55.376586    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:40:55.376891 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:55.376690    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls podName:a8752ff6-9b1b-4b50-8ece-1f045b1393f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:56.376669747 +0000 UTC m=+36.568507347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls") pod "image-registry-cb854fdc9-m6sx9" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4") : secret "image-registry-tls" not found
Sep 09 20:40:55.376891 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:55.376761    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls podName:5b863620-9ac7-4433-bba7-ed209457f8cf nodeName:}" failed. No retries permitted until 2026-09-09 20:40:56.376746523 +0000 UTC m=+36.568584101 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls") pod "dns-default-jhwgx" (UID: "5b863620-9ac7-4433-bba7-ed209457f8cf") : secret "dns-default-metrics-tls" not found
Sep 09 20:40:55.497680 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.497650    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 09 20:40:55.505263 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.505238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/78303679-3153-4d46-91c0-dd93139fa25d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:55.566113 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.566085    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 09 20:40:55.669635 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.669523    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 09 20:40:55.675585 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.675563    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/715ed8a6-4d2e-4169-8dc5-e72fe72850cd-klusterlet-config\") pod \"klusterlet-addon-workmgr-8bb9cfb9f-h4wx9\" (UID: \"715ed8a6-4d2e-4169-8dc5-e72fe72850cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:55.784013 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.783991    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 09 20:40:55.787489 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.787469    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmb6\" (UniqueName: \"kubernetes.io/projected/715ed8a6-4d2e-4169-8dc5-e72fe72850cd-kube-api-access-fdmb6\") pod \"klusterlet-addon-workmgr-8bb9cfb9f-h4wx9\" (UID: \"715ed8a6-4d2e-4169-8dc5-e72fe72850cd\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:55.794503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.794457    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8596x\" (UniqueName: \"kubernetes.io/projected/78303679-3153-4d46-91c0-dd93139fa25d-kube-api-access-8596x\") pod \"cluster-proxy-proxy-agent-847c5d47b4-9td79\" (UID: \"78303679-3153-4d46-91c0-dd93139fa25d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:55.794868 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.794842    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dplpx\" (UniqueName: \"kubernetes.io/projected/adb51729-08fc-4692-8ec2-be71226c33ec-kube-api-access-dplpx\") pod \"managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf\" (UID: \"adb51729-08fc-4692-8ec2-be71226c33ec\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"
Sep 09 20:40:55.867052 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.867020    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:40:55.884257 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.884232    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:40:55.892212 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:55.892195    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"
Sep 09 20:40:56.010682 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.010655    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rxdcl"]
Sep 09 20:40:56.141861 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.141837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"]
Sep 09 20:40:56.146448 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:56.146383    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715ed8a6_4d2e_4169_8dc5_e72fe72850cd.slice/crio-8ea376ed323c06093227ab903419d414c3e90eb31b5e051960ffc1a9f05e296d WatchSource:0}: Error finding container 8ea376ed323c06093227ab903419d414c3e90eb31b5e051960ffc1a9f05e296d: Status 404 returned error can't find the container with id 8ea376ed323c06093227ab903419d414c3e90eb31b5e051960ffc1a9f05e296d
Sep 09 20:40:56.146709 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:56.146658    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode562a150_9463_4b3d_a8be_a04e8703a450.slice/crio-92c0f2b527c6e8038fefef25281290fd00003cd069324865f8f67f5e49ead3c1 WatchSource:0}: Error finding container 92c0f2b527c6e8038fefef25281290fd00003cd069324865f8f67f5e49ead3c1: Status 404 returned error can't find the container with id 92c0f2b527c6e8038fefef25281290fd00003cd069324865f8f67f5e49ead3c1
Sep 09 20:40:56.280046 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.280011    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:40:56.280195 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.280014    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:40:56.280195 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.280178    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:40:56.282656 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.282623    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:40:56.282656 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.282645    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:40:56.282836 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.282754    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q6qnd\""
Sep 09 20:40:56.282907 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.282892    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:40:56.283009 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.282992    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sl49q\""
Sep 09 20:40:56.283163 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.283069    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:40:56.385043 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.385005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:56.385866 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.385077    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:56.385866 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:56.385160    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:40:56.385866 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:56.385207    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:40:56.385866 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:56.385223    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb854fdc9-m6sx9: secret "image-registry-tls" not found
Sep 09 20:40:56.385866 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:56.385228    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls podName:5b863620-9ac7-4433-bba7-ed209457f8cf nodeName:}" failed. No retries permitted until 2026-09-09 20:40:58.385207398 +0000 UTC m=+38.577044992 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls") pod "dns-default-jhwgx" (UID: "5b863620-9ac7-4433-bba7-ed209457f8cf") : secret "dns-default-metrics-tls" not found
Sep 09 20:40:56.385866 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:56.385273    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls podName:a8752ff6-9b1b-4b50-8ece-1f045b1393f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:40:58.385260289 +0000 UTC m=+38.577097866 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls") pod "image-registry-cb854fdc9-m6sx9" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4") : secret "image-registry-tls" not found
Sep 09 20:40:56.413171 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.413149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf"]
Sep 09 20:40:56.415677 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.415658    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"]
Sep 09 20:40:56.418609 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:56.418578    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadb51729_08fc_4692_8ec2_be71226c33ec.slice/crio-f8df0af75c331cbf1904328bde0ccf0a17dceee76cdf5cd67dd069a00d7523fd WatchSource:0}: Error finding container f8df0af75c331cbf1904328bde0ccf0a17dceee76cdf5cd67dd069a00d7523fd: Status 404 returned error can't find the container with id f8df0af75c331cbf1904328bde0ccf0a17dceee76cdf5cd67dd069a00d7523fd
Sep 09 20:40:56.419800 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:40:56.419776    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78303679_3153_4d46_91c0_dd93139fa25d.slice/crio-789d710ee92aeee4c6becd50efc32c8de31838b08e616616d5bec6ae9528bec2 WatchSource:0}: Error finding container 789d710ee92aeee4c6becd50efc32c8de31838b08e616616d5bec6ae9528bec2: Status 404 returned error can't find the container with id 789d710ee92aeee4c6becd50efc32c8de31838b08e616616d5bec6ae9528bec2
Sep 09 20:40:56.465319 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.465274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rxdcl" event={"ID":"e562a150-9463-4b3d-a8be-a04e8703a450","Type":"ContainerStarted","Data":"92c0f2b527c6e8038fefef25281290fd00003cd069324865f8f67f5e49ead3c1"}
Sep 09 20:40:56.466549 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.466526    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9" event={"ID":"715ed8a6-4d2e-4169-8dc5-e72fe72850cd","Type":"ContainerStarted","Data":"8ea376ed323c06093227ab903419d414c3e90eb31b5e051960ffc1a9f05e296d"}
Sep 09 20:40:56.469342 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.469301    2566 generic.go:356] "Generic (PLEG): container finished" podID="b472e626-f237-4a9a-a307-fc33a70a570f" containerID="339d1da4c26043bbe542802d4acaa4a2d718683b4a5dc3596c44b514df61a6cb" exitCode=0
Sep 09 20:40:56.469441 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.469395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2vzw" event={"ID":"b472e626-f237-4a9a-a307-fc33a70a570f","Type":"ContainerDied","Data":"339d1da4c26043bbe542802d4acaa4a2d718683b4a5dc3596c44b514df61a6cb"}
Sep 09 20:40:56.470527 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.470508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf" event={"ID":"adb51729-08fc-4692-8ec2-be71226c33ec","Type":"ContainerStarted","Data":"f8df0af75c331cbf1904328bde0ccf0a17dceee76cdf5cd67dd069a00d7523fd"}
Sep 09 20:40:56.471614 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:56.471592    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" event={"ID":"78303679-3153-4d46-91c0-dd93139fa25d","Type":"ContainerStarted","Data":"789d710ee92aeee4c6becd50efc32c8de31838b08e616616d5bec6ae9528bec2"}
Sep 09 20:40:57.486814 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:57.486563    2566 generic.go:356] "Generic (PLEG): container finished" podID="b472e626-f237-4a9a-a307-fc33a70a570f" containerID="0e0638671c26fca2e1718b0c0f9f3bdaf8cc3eb75f635df12829268f6a4aba1a" exitCode=0
Sep 09 20:40:57.487375 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:57.486835    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2vzw" event={"ID":"b472e626-f237-4a9a-a307-fc33a70a570f","Type":"ContainerDied","Data":"0e0638671c26fca2e1718b0c0f9f3bdaf8cc3eb75f635df12829268f6a4aba1a"}
Sep 09 20:40:57.914395 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:57.914358    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:58.016537 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:58.016464    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:58.405883 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:58.405788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:40:58.406055 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:58.405905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:40:58.406055 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:58.406032    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:40:58.406055 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:58.406048    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb854fdc9-m6sx9: secret "image-registry-tls" not found
Sep 09 20:40:58.406209 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:58.406031    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:40:58.406209 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:58.406107    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls podName:a8752ff6-9b1b-4b50-8ece-1f045b1393f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:41:02.406086833 +0000 UTC m=+42.597924429 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls") pod "image-registry-cb854fdc9-m6sx9" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4") : secret "image-registry-tls" not found
Sep 09 20:40:58.406209 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:40:58.406183    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls podName:5b863620-9ac7-4433-bba7-ed209457f8cf nodeName:}" failed. No retries permitted until 2026-09-09 20:41:02.406162218 +0000 UTC m=+42.597999800 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls") pod "dns-default-jhwgx" (UID: "5b863620-9ac7-4433-bba7-ed209457f8cf") : secret "dns-default-metrics-tls" not found
Sep 09 20:40:58.499234 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:58.499195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-r2vzw" event={"ID":"b472e626-f237-4a9a-a307-fc33a70a570f","Type":"ContainerStarted","Data":"97511ae98f644edba1633782b7cff817abdd1c22bd76f7d3d0339cb2024e0a80"}
Sep 09 20:40:58.522863 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:58.522125    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-r2vzw" podStartSLOduration=7.639926945 podStartE2EDuration="38.522101951s" totalImagesPullingTime="30.882175006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:40:58.51833793 +0000 UTC m=+38.710175530" watchObservedRunningTime="2026-09-09 20:40:58.522101951 +0000 UTC m=+38.713939552"
Sep 09 20:40:58.742125 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:58.742089    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:58.810803 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:58.810773    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:59.221896 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:59.221760    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:59.321877 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:59.321838    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:40:59.423043 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:40:59.423005    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:00.171865 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:00.171825    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:00.234530 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:00.234492    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:02.440115 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:02.440078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:41:02.440553 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:02.440134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:41:02.440553 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:02.440224    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:41:02.440553 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:02.440265    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:41:02.440553 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:02.440289    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb854fdc9-m6sx9: secret "image-registry-tls" not found
Sep 09 20:41:02.440553 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:02.440280    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls podName:5b863620-9ac7-4433-bba7-ed209457f8cf nodeName:}" failed. No retries permitted until 2026-09-09 20:41:10.440266403 +0000 UTC m=+50.632103980 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls") pod "dns-default-jhwgx" (UID: "5b863620-9ac7-4433-bba7-ed209457f8cf") : secret "dns-default-metrics-tls" not found
Sep 09 20:41:02.440553 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:02.440374    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls podName:a8752ff6-9b1b-4b50-8ece-1f045b1393f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:41:10.440356297 +0000 UTC m=+50.632193881 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls") pod "image-registry-cb854fdc9-m6sx9" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4") : secret "image-registry-tls" not found
Sep 09 20:41:03.713519 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:03.713479    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:03.787023 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:03.786983    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:04.706027 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:04.705995    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:04.805731 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:04.805698    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:05.519430 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.519392    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf" event={"ID":"adb51729-08fc-4692-8ec2-be71226c33ec","Type":"ContainerStarted","Data":"b3e7c2c483ad3d73a1a0f5b2615985d43e95e28acaa493b0e85d3d6c29f28313"}
Sep 09 20:41:05.521589 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.521564    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" event={"ID":"78303679-3153-4d46-91c0-dd93139fa25d","Type":"ContainerStarted","Data":"c6f01f664e3e45f4fe3ff5f25a4159b571f18fd18919b25bf8ed66ec0a166e7b"}
Sep 09 20:41:05.521709 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.521600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" event={"ID":"78303679-3153-4d46-91c0-dd93139fa25d","Type":"ContainerStarted","Data":"e20812b6d986f77ac7a96cabc8dcf4ab7f12f3accba5b8132a9b6b3dc78ff394"}
Sep 09 20:41:05.521709 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.521618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" event={"ID":"78303679-3153-4d46-91c0-dd93139fa25d","Type":"ContainerStarted","Data":"634e387dcc3837b3cdf1c0495900f36188d0b3894a459741efa7793e94dc0202"}
Sep 09 20:41:05.523126 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.523104    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rxdcl" event={"ID":"e562a150-9463-4b3d-a8be-a04e8703a450","Type":"ContainerStarted","Data":"9dc98df7c06f58662942b65386ef6260c6ea5f0dfdd6fd775456651dd17993c0"}
Sep 09 20:41:05.524460 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.524441    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9" event={"ID":"715ed8a6-4d2e-4169-8dc5-e72fe72850cd","Type":"ContainerStarted","Data":"2cec7c69a58b313f016fbe3d10392689751f7eb26f03a105f9dd5f497cdaf8ef"}
Sep 09 20:41:05.524652 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.524636    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:41:05.526184 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.526168    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9"
Sep 09 20:41:05.536553 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.536517    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-7b7f6c74fd-rvhbf" podStartSLOduration=13.991644303 podStartE2EDuration="22.536489401s" totalImagesPullingTime="8.544845098s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:05.53580836 +0000 UTC m=+45.727645959" watchObservedRunningTime="2026-09-09 20:41:05.536489401 +0000 UTC m=+45.728326997"
Sep 09 20:41:05.554265 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.554206    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rxdcl" podStartSLOduration=2.745245899 podStartE2EDuration="11.554193444s" totalImagesPullingTime="8.808947545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:05.553398806 +0000 UTC m=+45.745236415" watchObservedRunningTime="2026-09-09 20:41:05.554193444 +0000 UTC m=+45.746031043"
Sep 09 20:41:05.571640 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.571595    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" podStartSLOduration=14.037564444 podStartE2EDuration="22.571579269s" totalImagesPullingTime="8.534014825s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:05.570333423 +0000 UTC m=+45.762171015" watchObservedRunningTime="2026-09-09 20:41:05.571579269 +0000 UTC m=+45.763416868"
Sep 09 20:41:05.585423 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:05.585371    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-8bb9cfb9f-h4wx9" podStartSLOduration=13.776777059 podStartE2EDuration="22.585359658s" totalImagesPullingTime="8.808582599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:05.58474712 +0000 UTC m=+45.776584719" watchObservedRunningTime="2026-09-09 20:41:05.585359658 +0000 UTC m=+45.777197257"
Sep 09 20:41:07.232533 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:07.232500    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:07.334074 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:07.334042    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:07.391826 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:07.391804    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:10.503477 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:10.503437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:41:10.503851 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:10.503500    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:41:10.503851 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:10.503585    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:41:10.503851 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:10.503609    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb854fdc9-m6sx9: secret "image-registry-tls" not found
Sep 09 20:41:10.503851 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:10.503594    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:41:10.503851 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:10.503685    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls podName:a8752ff6-9b1b-4b50-8ece-1f045b1393f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:41:26.503665878 +0000 UTC m=+66.695503470 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls") pod "image-registry-cb854fdc9-m6sx9" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4") : secret "image-registry-tls" not found
Sep 09 20:41:10.503851 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:10.503714    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls podName:5b863620-9ac7-4433-bba7-ed209457f8cf nodeName:}" failed. No retries permitted until 2026-09-09 20:41:26.503702031 +0000 UTC m=+66.695539608 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls") pod "dns-default-jhwgx" (UID: "5b863620-9ac7-4433-bba7-ed209457f8cf") : secret "dns-default-metrics-tls" not found
Sep 09 20:41:11.342662 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:11.342631    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:11.404602 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:11.404574    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:11.528274 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:11.528240    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:15.252359 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:15.252325    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:15.329807 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:15.329774    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:15.451539 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:15.451506    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:16.400083 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:16.400052    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:16.461335 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:16.461291    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:18.791083 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:18.791051    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:18.892666 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:18.892634    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:18.961098 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:18.961070    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:20.458606 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:20.458579    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6n527"
Sep 09 20:41:20.627562 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:20.627531    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:20.725015 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:20.724929    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:21.413740 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:21.413708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:21.511833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:21.511800    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:21.758926 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:21.758896    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:21.860282 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:21.860247    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:23.776567 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:23.776538    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:23.836408 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:23.836374    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:25.010273 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.010236    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:41:25.012417 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.012396    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 09 20:41:25.023247 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.023216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2fe41335-60ef-4123-99c6-0c504cf31555-original-pull-secret\") pod \"global-pull-secret-syncer-zm7nt\" (UID: \"2fe41335-60ef-4123-99c6-0c504cf31555\") " pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:41:25.089942 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.089914    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zm7nt"
Sep 09 20:41:25.111080 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.111052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:41:25.113553 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.113534    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 09 20:41:25.122198 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:25.122177    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 09 20:41:25.122291 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:25.122244    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs podName:cad3822b-6c99-478f-a17a-26ca43f5fd7e nodeName:}" failed. No retries permitted until 2026-09-09 20:42:29.122227039 +0000 UTC m=+129.314064616 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs") pod "network-metrics-daemon-qk9rl" (UID: "cad3822b-6c99-478f-a17a-26ca43f5fd7e") : secret "metrics-daemon-secret" not found
Sep 09 20:41:25.212242 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.212212    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:41:25.215036 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.215018    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 09 20:41:25.224983 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.224965    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 09 20:41:25.235158 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.235134    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pdpl\" (UniqueName: \"kubernetes.io/projected/cac22caa-f302-4b8e-bb6e-5eef356214b2-kube-api-access-9pdpl\") pod \"network-check-target-8rk4n\" (UID: \"cac22caa-f302-4b8e-bb6e-5eef356214b2\") " pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:41:25.360510 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.360438    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zm7nt"]
Sep 09 20:41:25.365128 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:41:25.365105    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fe41335_60ef_4123_99c6_0c504cf31555.slice/crio-49f137f539c86bccf2935ab1d9ff88db9aa603417cc0af001be69c43a191fa0d WatchSource:0}: Error finding container 49f137f539c86bccf2935ab1d9ff88db9aa603417cc0af001be69c43a191fa0d: Status 404 returned error can't find the container with id 49f137f539c86bccf2935ab1d9ff88db9aa603417cc0af001be69c43a191fa0d
Sep 09 20:41:25.397663 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.397644    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-q6qnd\""
Sep 09 20:41:25.405404 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.405388    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:41:25.579499 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.579460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zm7nt" event={"ID":"2fe41335-60ef-4123-99c6-0c504cf31555","Type":"ContainerStarted","Data":"49f137f539c86bccf2935ab1d9ff88db9aa603417cc0af001be69c43a191fa0d"}
Sep 09 20:41:25.698645 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:25.698576    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8rk4n"]
Sep 09 20:41:25.701903 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:41:25.701884    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac22caa_f302_4b8e_bb6e_5eef356214b2.slice/crio-6315b5f1dcecb7da519f2d781fd01df64a24cb9e7355117008ed008359137c38 WatchSource:0}: Error finding container 6315b5f1dcecb7da519f2d781fd01df64a24cb9e7355117008ed008359137c38: Status 404 returned error can't find the container with id 6315b5f1dcecb7da519f2d781fd01df64a24cb9e7355117008ed008359137c38
Sep 09 20:41:26.522107 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.522072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:41:26.522587 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.522153    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:41:26.522587 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:26.522251    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 09 20:41:26.522587 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:26.522297    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 09 20:41:26.522587 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:26.522350    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-cb854fdc9-m6sx9: secret "image-registry-tls" not found
Sep 09 20:41:26.522587 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:26.522337    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls podName:5b863620-9ac7-4433-bba7-ed209457f8cf nodeName:}" failed. No retries permitted until 2026-09-09 20:41:58.522302965 +0000 UTC m=+98.714140545 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls") pod "dns-default-jhwgx" (UID: "5b863620-9ac7-4433-bba7-ed209457f8cf") : secret "dns-default-metrics-tls" not found
Sep 09 20:41:26.522587 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:26.522439    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls podName:a8752ff6-9b1b-4b50-8ece-1f045b1393f4 nodeName:}" failed. No retries permitted until 2026-09-09 20:41:58.522421207 +0000 UTC m=+98.714258793 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls") pod "image-registry-cb854fdc9-m6sx9" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4") : secret "image-registry-tls" not found
Sep 09 20:41:26.540690 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.540358    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-kg5pd"]
Sep 09 20:41:26.554087 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.553845    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.556367 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.556289    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kg5pd"]
Sep 09 20:41:26.556837 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.556551    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 09 20:41:26.556837 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.556667    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-r8hx4\""
Sep 09 20:41:26.558331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.557111    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 09 20:41:26.558331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.557181    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 09 20:41:26.558331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.557342    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 09 20:41:26.584674 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.584645    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8rk4n" event={"ID":"cac22caa-f302-4b8e-bb6e-5eef356214b2","Type":"ContainerStarted","Data":"6315b5f1dcecb7da519f2d781fd01df64a24cb9e7355117008ed008359137c38"}
Sep 09 20:41:26.589068 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.589035    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:26.623124 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.622961    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-crio-socket\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.623124 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.623016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c69b7\" (UniqueName: \"kubernetes.io/projected/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-api-access-c69b7\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.623124 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.623058    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.623124 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.623090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-data-volume\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.623421 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.623155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.645398 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.645327    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:26.723739 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.723714    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c69b7\" (UniqueName: \"kubernetes.io/projected/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-api-access-c69b7\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.723876 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.723757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.723876 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.723779    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-data-volume\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.723876 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.723822    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.724035 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:26.723873    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:26.724035 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.723887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-crio-socket\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.724035 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:26.723934    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:27.223916077 +0000 UTC m=+67.415753677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:26.724035 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.723972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-crio-socket\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.724035 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:26.724028    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:41:26.724271 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:26.724065    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:27.224055134 +0000 UTC m=+67.415892714 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : configmap "kube-rbac-proxy" not found
Sep 09 20:41:26.724271 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.724234    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-data-volume\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:26.734126 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:26.734103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c69b7\" (UniqueName: \"kubernetes.io/projected/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-api-access-c69b7\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:27.227265 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:27.227234    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:27.227512 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:27.227340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:27.227512 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:27.227375    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:41:27.227512 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:27.227439    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:28.227424671 +0000 UTC m=+68.419262247 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : configmap "kube-rbac-proxy" not found
Sep 09 20:41:27.227512 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:27.227443    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:27.227512 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:27.227486    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:28.227472041 +0000 UTC m=+68.419309618 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:27.948107 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:27.948050    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:28.234808 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:28.234765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:28.234984 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:28.234922    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:28.235216 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:28.235129    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:41:28.235216 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:28.235143    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:28.235458 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:28.235224    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:30.235200537 +0000 UTC m=+70.427038124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:28.235458 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:28.235250    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:30.235240043 +0000 UTC m=+70.427077621 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : configmap "kube-rbac-proxy" not found
Sep 09 20:41:28.376292 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:28.376248    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:28.487822 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:28.487742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:28.580630 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:28.580594    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:28.681162 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:28.681124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:28.712776 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:28.712750    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qdsbk_f60f0088-211f-490c-8012-edec87817ca1/dns-node-resolver/0.log"
Sep 09 20:41:28.782797 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:28.782709    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:29.712908 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:29.712879    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g2d92_28f00347-ee67-4e91-ac2f-0dbe68892960/node-ca/0.log"
Sep 09 20:41:30.250883 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:30.250851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:30.251072 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:30.250911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:30.251072 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:30.251007    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:30.251072 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:30.251028    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:41:30.251232 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:30.251083    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:34.251066596 +0000 UTC m=+74.442904178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : configmap "kube-rbac-proxy" not found
Sep 09 20:41:30.251232 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:30.251098    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:34.251091527 +0000 UTC m=+74.442929103 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:30.288063 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:30.288031    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:30.511948 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:30.511888    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rxdcl_e562a150-9463-4b3d-a8be-a04e8703a450/serve-healthcheck-canary/0.log"
Sep 09 20:41:30.599964 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:30.599925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8rk4n" event={"ID":"cac22caa-f302-4b8e-bb6e-5eef356214b2","Type":"ContainerStarted","Data":"9d6844b68c9526c7ce64e73a4d0b0925c06eb728fb504a6220c039efa9bfc46a"}
Sep 09 20:41:30.600742 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:30.600716    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:41:30.602743 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:30.602715    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zm7nt" event={"ID":"2fe41335-60ef-4123-99c6-0c504cf31555","Type":"ContainerStarted","Data":"1f0b8709911644c1dcd634aebc2a881e07388742f4c97178db971e09de98556c"}
Sep 09 20:41:30.619507 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:30.619466    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8rk4n" podStartSLOduration=65.953162313 podStartE2EDuration="1m10.619452744s" totalImagesPullingTime="4.666290431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:30.618787734 +0000 UTC m=+70.810625332" watchObservedRunningTime="2026-09-09 20:41:30.619452744 +0000 UTC m=+70.811290340"
Sep 09 20:41:30.637855 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:30.637815    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zm7nt" podStartSLOduration=65.63096583 podStartE2EDuration="1m10.637803365s" totalImagesPullingTime="5.006837535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:30.636934575 +0000 UTC m=+70.828772186" watchObservedRunningTime="2026-09-09 20:41:30.637803365 +0000 UTC m=+70.829640963"
Sep 09 20:41:31.270848 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:31.270819    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:31.968649 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:31.968620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:32.068789 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:32.068756    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:32.187416 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:32.187384    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:32.283491 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:32.283463    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:32.384956 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:32.384924    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:32.485106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:32.485079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:32.662406 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:32.662331    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:33.466808 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:33.466771    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:33.860636 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:33.860602    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:34.061219 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:34.061187    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:34.262607 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:34.262581    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:34.279577 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:34.279558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:34.279658 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:34.279596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:34.279701 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:34.279693    2566 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 09 20:41:34.279733 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:34.279701    2566 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:34.279774 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:34.279748    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:42.279734485 +0000 UTC m=+82.471572065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : secret "insights-runtime-extractor-tls" not found
Sep 09 20:41:34.279774 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:41:34.279762    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm podName:ac16bb0c-a835-4de8-aa64-0b9d85c2508c nodeName:}" failed. No retries permitted until 2026-09-09 20:41:42.279755395 +0000 UTC m=+82.471592972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-kg5pd" (UID: "ac16bb0c-a835-4de8-aa64-0b9d85c2508c") : configmap "kube-rbac-proxy" not found
Sep 09 20:41:34.777403 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:34.777371    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:35.270094 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:35.270070    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:35.458655 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:35.458620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:35.837050 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:35.837016    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:35.888624 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:35.888599    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:36.048778 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:36.048750    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:36.511775 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:36.511738    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:36.568348 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:36.568302    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:37.059798 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:37.059768    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:37.457254 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:37.457183    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:37.654716 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:37.654683    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:38.253918 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:38.253885    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:38.353706 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:38.353672    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:38.652689 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:38.652609    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:39.055965 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:39.055930    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:39.265636 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:39.265611    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:39.667446 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:39.667421    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:39.768564 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:39.768537    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:40.453221 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:40.453191    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:40.660080 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:40.660042    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:40.853162 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:40.853139    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:41.252398 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:41.252367    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:41.854214 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:41.854182    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:42.257903 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.257866    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:42.336800 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.336774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:42.336959 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.336819    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:42.337296 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.337278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:42.339028 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.339005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ac16bb0c-a835-4de8-aa64-0b9d85c2508c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-kg5pd\" (UID: \"ac16bb0c-a835-4de8-aa64-0b9d85c2508c\") " pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:42.454991 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.454948    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:42.466763 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.466743    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-kg5pd"
Sep 09 20:41:42.576648 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.576546    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:42.656170 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.656141    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:42.732676 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:42.732641    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-kg5pd"]
Sep 09 20:41:42.735687 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:41:42.735661    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac16bb0c_a835_4de8_aa64_0b9d85c2508c.slice/crio-08d42057a21a4613486fdb8c5ee15cf7ce472b689654b79e3e2f23cb36e33045 WatchSource:0}: Error finding container 08d42057a21a4613486fdb8c5ee15cf7ce472b689654b79e3e2f23cb36e33045: Status 404 returned error can't find the container with id 08d42057a21a4613486fdb8c5ee15cf7ce472b689654b79e3e2f23cb36e33045
Sep 09 20:41:43.641759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:43.641732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kg5pd" event={"ID":"ac16bb0c-a835-4de8-aa64-0b9d85c2508c","Type":"ContainerStarted","Data":"98e304c0bc8c8c19163ed170d4bcaffdf35e73fdd2c902ce94f8f65388f7538f"}
Sep 09 20:41:43.642064 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:43.641782    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kg5pd" event={"ID":"ac16bb0c-a835-4de8-aa64-0b9d85c2508c","Type":"ContainerStarted","Data":"a013b331c0fa91d010daa54e7865cd266dcbaed045518358affd83375be19320"}
Sep 09 20:41:43.642064 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:43.641801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kg5pd" event={"ID":"ac16bb0c-a835-4de8-aa64-0b9d85c2508c","Type":"ContainerStarted","Data":"08d42057a21a4613486fdb8c5ee15cf7ce472b689654b79e3e2f23cb36e33045"}
Sep 09 20:41:45.649130 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:45.649093    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-kg5pd" event={"ID":"ac16bb0c-a835-4de8-aa64-0b9d85c2508c","Type":"ContainerStarted","Data":"af0766ef1caa523a94b896cceb0e6bb55bc6af34c09b46a7f9195ef569e1c4b0"}
Sep 09 20:41:45.671696 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:45.671653    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-kg5pd" podStartSLOduration=17.477384656 podStartE2EDuration="19.671639151s" totalImagesPullingTime="2.194254495s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 20:41:26 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:45.670984766 +0000 UTC m=+85.862822364" watchObservedRunningTime="2026-09-09 20:41:45.671639151 +0000 UTC m=+85.863476750"
Sep 09 20:41:47.430852 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:47.430819    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:47.543881 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:47.543848    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:47.759524 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:47.759481    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:48.112145 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:48.112064    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:48.168376 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:48.168347    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:48.518934 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:48.518896    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:48.576044 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:48.576018    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:48.721219 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:48.721186    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:49.325789 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:49.325756    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:49.523785 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:49.523752    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:50.202140 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:50.202103    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:50.350697 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:50.350665    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:50.448215 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:50.448186    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:51.412215 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:51.412181    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:51.505382 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:51.505349    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:51.600613 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:51.600578    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:51.703119 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:51.703050    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:51.811652 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:51.811610    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:51.911793 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:51.911763    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:52.229367 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:52.229298    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:52.293116 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:52.293075    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:54.348411 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:54.348375    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:55.404368 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:55.404335    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:55.460723 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:55.460685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:55.960911 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:55.960881    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:56.059965 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:56.059934    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:56.137947 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:56.137920    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:56.545784 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:56.545753    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:56.679579 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:56.679543    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:41:58.561633 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:58.561598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:41:58.561633 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:58.561630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:41:58.563756 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:58.563734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5b863620-9ac7-4433-bba7-ed209457f8cf-metrics-tls\") pod \"dns-default-jhwgx\" (UID: \"5b863620-9ac7-4433-bba7-ed209457f8cf\") " pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:41:58.563989 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:58.563969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"image-registry-cb854fdc9-m6sx9\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") " pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:41:58.621283 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:58.621262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vs5mb\""
Sep 09 20:41:58.629588 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:58.629570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:41:58.860591 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:58.860514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-tj77j\""
Sep 09 20:41:58.868483 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:58.868462    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:41:58.900895 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:58.900868    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-jhwgx"]
Sep 09 20:41:58.904284 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:41:58.904260    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b863620_9ac7_4433_bba7_ed209457f8cf.slice/crio-2944ac83b9852c2eb3b2cfa4188cec83e9dc1ddcfdb60ff04459196aaac5582b WatchSource:0}: Error finding container 2944ac83b9852c2eb3b2cfa4188cec83e9dc1ddcfdb60ff04459196aaac5582b: Status 404 returned error can't find the container with id 2944ac83b9852c2eb3b2cfa4188cec83e9dc1ddcfdb60ff04459196aaac5582b
Sep 09 20:41:59.140376 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:59.140301    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-cb854fdc9-m6sx9"]
Sep 09 20:41:59.143741 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:41:59.143715    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8752ff6_9b1b_4b50_8ece_1f045b1393f4.slice/crio-ced81d8d7d88fa1da22639b0b93c6cb81a81794ffdb840c3cd89dfa02b196a2d WatchSource:0}: Error finding container ced81d8d7d88fa1da22639b0b93c6cb81a81794ffdb840c3cd89dfa02b196a2d: Status 404 returned error can't find the container with id ced81d8d7d88fa1da22639b0b93c6cb81a81794ffdb840c3cd89dfa02b196a2d
Sep 09 20:41:59.690004 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:59.689889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9" event={"ID":"a8752ff6-9b1b-4b50-8ece-1f045b1393f4","Type":"ContainerStarted","Data":"7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6"}
Sep 09 20:41:59.690004 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:59.689949    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9" event={"ID":"a8752ff6-9b1b-4b50-8ece-1f045b1393f4","Type":"ContainerStarted","Data":"ced81d8d7d88fa1da22639b0b93c6cb81a81794ffdb840c3cd89dfa02b196a2d"}
Sep 09 20:41:59.690004 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:59.689975    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:41:59.691346 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:59.691323    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhwgx" event={"ID":"5b863620-9ac7-4433-bba7-ed209457f8cf","Type":"ContainerStarted","Data":"2944ac83b9852c2eb3b2cfa4188cec83e9dc1ddcfdb60ff04459196aaac5582b"}
Sep 09 20:41:59.709957 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:41:59.709906    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9" podStartSLOduration=99.709887695 podStartE2EDuration="1m39.709887695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:41:59.708140729 +0000 UTC m=+99.899978329" watchObservedRunningTime="2026-09-09 20:41:59.709887695 +0000 UTC m=+99.901725296"
Sep 09 20:42:01.140185 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:01.140151    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:01.206250 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:01.206210    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:01.483576 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:01.483543    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:01.699453 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:01.699420    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhwgx" event={"ID":"5b863620-9ac7-4433-bba7-ed209457f8cf","Type":"ContainerStarted","Data":"868b93e93d77a76098f61e275208de3dc3f9699daa2bb35ed605c0a5169fe283"}
Sep 09 20:42:01.699597 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:01.699477    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-jhwgx" event={"ID":"5b863620-9ac7-4433-bba7-ed209457f8cf","Type":"ContainerStarted","Data":"bc858e154f594eabcd0937203a30b11192a9dfc711f8e6ab33e841f9719f1a28"}
Sep 09 20:42:01.699597 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:01.699552    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:42:02.527276 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:02.527237    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:02.589746 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:02.589709    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:02.611924 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:02.611901    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8rk4n"
Sep 09 20:42:02.627880 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:02.627839    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-jhwgx" podStartSLOduration=66.571939772 podStartE2EDuration="1m8.627827257s" totalImagesPullingTime="2.055887485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:01.719506567 +0000 UTC m=+101.911344167" watchObservedRunningTime="2026-09-09 20:42:02.627827257 +0000 UTC m=+102.819664856"
Sep 09 20:42:02.950912 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:02.950820    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:03.010579 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:03.010549    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:03.944433 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:03.944401    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:04.003349 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:04.003304    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:04.433906 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:04.433878    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:06.287267 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.287237    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:06.423825 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.423794    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:06.525893 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.525857    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:06.571533 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.571458    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ngv59"]
Sep 09 20:42:06.574249 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.574233    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.580154 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.579869    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 09 20:42:06.580518 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.580500    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dvqdx\""
Sep 09 20:42:06.580626 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.580501    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 09 20:42:06.580626 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.580501    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 09 20:42:06.580626 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.580545    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 09 20:42:06.580626 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.580570    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 09 20:42:06.580887 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.580877    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 09 20:42:06.624882 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.624847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:06.725051 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.725016    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d273d6b-ed54-4130-9680-f0b93a3dd791-sys\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.725252 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.725061    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-textfile\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.725252 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.725087    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.725252 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.725115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-accelerators-collector-config\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.725252 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.725160    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmbn2\" (UniqueName: \"kubernetes.io/projected/2d273d6b-ed54-4130-9680-f0b93a3dd791-kube-api-access-hmbn2\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.725252 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.725207    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-wtmp\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.725252 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.725236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d273d6b-ed54-4130-9680-f0b93a3dd791-metrics-client-ca\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.725494 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.725262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d273d6b-ed54-4130-9680-f0b93a3dd791-root\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.725494 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.725345    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-tls\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.749731 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.749695    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:06.826075 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.825991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d273d6b-ed54-4130-9680-f0b93a3dd791-sys\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826075 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826036    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-textfile\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826075 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826061    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826355 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826087    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-accelerators-collector-config\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826355 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826111    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/2d273d6b-ed54-4130-9680-f0b93a3dd791-sys\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826355 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826121    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmbn2\" (UniqueName: \"kubernetes.io/projected/2d273d6b-ed54-4130-9680-f0b93a3dd791-kube-api-access-hmbn2\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826355 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-wtmp\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826355 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826219    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d273d6b-ed54-4130-9680-f0b93a3dd791-metrics-client-ca\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826588 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d273d6b-ed54-4130-9680-f0b93a3dd791-root\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826588 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826382    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-wtmp\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826588 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-tls\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826588 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/2d273d6b-ed54-4130-9680-f0b93a3dd791-root\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826588 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826424    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-textfile\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826588 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:42:06.826519    2566 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 09 20:42:06.826588 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:42:06.826583    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-tls podName:2d273d6b-ed54-4130-9680-f0b93a3dd791 nodeName:}" failed. No retries permitted until 2026-09-09 20:42:07.32656575 +0000 UTC m=+107.518403341 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-tls") pod "node-exporter-ngv59" (UID: "2d273d6b-ed54-4130-9680-f0b93a3dd791") : secret "node-exporter-tls" not found
Sep 09 20:42:06.826826 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826752    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-accelerators-collector-config\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.826826 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.826808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2d273d6b-ed54-4130-9680-f0b93a3dd791-metrics-client-ca\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.828507 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.828491    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.836724 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.836704    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmbn2\" (UniqueName: \"kubernetes.io/projected/2d273d6b-ed54-4130-9680-f0b93a3dd791-kube-api-access-hmbn2\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:06.871136 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.871107    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:06.971550 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:06.971512    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:07.073660 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.073631    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:07.171282 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.171201    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:07.330591 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.330560    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-tls\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:07.332999 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.332975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/2d273d6b-ed54-4130-9680-f0b93a3dd791-node-exporter-tls\") pod \"node-exporter-ngv59\" (UID: \"2d273d6b-ed54-4130-9680-f0b93a3dd791\") " pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:07.356324 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.356278    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:07.483249 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.483224    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ngv59"
Sep 09 20:42:07.492821 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:42:07.492791    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d273d6b_ed54_4130_9680_f0b93a3dd791.slice/crio-74338d846f0097ba2763a41aeba0d60b6af0c367aa11511a01109367b38e4922 WatchSource:0}: Error finding container 74338d846f0097ba2763a41aeba0d60b6af0c367aa11511a01109367b38e4922: Status 404 returned error can't find the container with id 74338d846f0097ba2763a41aeba0d60b6af0c367aa11511a01109367b38e4922
Sep 09 20:42:07.514164 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.514135    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:07.609291 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.609259    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:07.722375 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.722337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngv59" event={"ID":"2d273d6b-ed54-4130-9680-f0b93a3dd791","Type":"ContainerStarted","Data":"74338d846f0097ba2763a41aeba0d60b6af0c367aa11511a01109367b38e4922"}
Sep 09 20:42:07.829422 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.829336    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:07.925518 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:07.925478    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:09.548664 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:09.548629    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:09.623529 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:09.623497    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:09.730325 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:09.730286    2566 generic.go:356] "Generic (PLEG): container finished" podID="2d273d6b-ed54-4130-9680-f0b93a3dd791" containerID="a88f18efcf49335c05ea74451e15f5583204d462c2f070dd686953952a941889" exitCode=0
Sep 09 20:42:09.730497 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:09.730374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngv59" event={"ID":"2d273d6b-ed54-4130-9680-f0b93a3dd791","Type":"ContainerDied","Data":"a88f18efcf49335c05ea74451e15f5583204d462c2f070dd686953952a941889"}
Sep 09 20:42:10.044435 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:10.044405    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:10.166667 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:10.166561    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:10.735275 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:10.735238    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngv59" event={"ID":"2d273d6b-ed54-4130-9680-f0b93a3dd791","Type":"ContainerStarted","Data":"b2895d1acf914ec8cccf4608f62787037affeb5f339e202c7ba562073b3e9598"}
Sep 09 20:42:10.735650 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:10.735350    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ngv59" event={"ID":"2d273d6b-ed54-4130-9680-f0b93a3dd791","Type":"ContainerStarted","Data":"2732e09f3050befe9516b674d8743da431aa8cf2fda0717a65537dc0e3295e19"}
Sep 09 20:42:10.753048 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:10.753002    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ngv59" podStartSLOduration=3.502204701 podStartE2EDuration="4.752986852s" totalImagesPullingTime="1.188245622s" totalInitContainerRuntime="62.536529ms" isStatefulPod=true podCreationTimestamp="2026-09-09 20:42:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:10.751924473 +0000 UTC m=+110.943762088" watchObservedRunningTime="2026-09-09 20:42:10.752986852 +0000 UTC m=+110.944824450"
Sep 09 20:42:11.345845 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:11.345803    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:11.414698 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:11.414657    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:11.606506 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:11.606422    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:11.703748 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:11.703707    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:11.704627 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:11.704605    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-jhwgx"
Sep 09 20:42:11.824535 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:11.824501    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:11.920524 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:11.920441    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:12.577275 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:12.577234    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:12.637280 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:12.637246    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:12.758887 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:12.758860    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:13.326502 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:13.326472    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb854fdc9-m6sx9"]
Sep 09 20:42:13.404689 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:13.404654    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:13.513871 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:13.513839    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:13.605450 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:13.605352    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:15.885581 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:15.885533    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" podUID="78303679-3153-4d46-91c0-dd93139fa25d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 20:42:16.358910 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:16.358862    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:16.459449 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:16.459414    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:22.226245 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:22.226204    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:22.308987 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:22.308956    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:23.151697 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:23.151660    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:23.252806 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:23.252776    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:23.312859 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:23.312822    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:23.332406 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:23.332378    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:42:24.393038 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:24.393003    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:25.110806 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:25.110767    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:25.212760 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:25.212722    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:25.886050 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:25.886012    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" podUID="78303679-3153-4d46-91c0-dd93139fa25d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 20:42:29.189048 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:29.189007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:42:29.191212 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:29.191183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cad3822b-6c99-478f-a17a-26ca43f5fd7e-metrics-certs\") pod \"network-metrics-daemon-qk9rl\" (UID: \"cad3822b-6c99-478f-a17a-26ca43f5fd7e\") " pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:42:29.302251 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:29.302218    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-sl49q\""
Sep 09 20:42:29.310963 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:29.310942    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qk9rl"
Sep 09 20:42:29.576908 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:29.576879    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qk9rl"]
Sep 09 20:42:29.580926 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:42:29.580899    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad3822b_6c99_478f_a17a_26ca43f5fd7e.slice/crio-dfa8431886bae9338617a8cd15c9805946fe2777d4b49e3e5ba252ca4f77f409 WatchSource:0}: Error finding container dfa8431886bae9338617a8cd15c9805946fe2777d4b49e3e5ba252ca4f77f409: Status 404 returned error can't find the container with id dfa8431886bae9338617a8cd15c9805946fe2777d4b49e3e5ba252ca4f77f409
Sep 09 20:42:29.794359 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:29.794331    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qk9rl" event={"ID":"cad3822b-6c99-478f-a17a-26ca43f5fd7e","Type":"ContainerStarted","Data":"dfa8431886bae9338617a8cd15c9805946fe2777d4b49e3e5ba252ca4f77f409"}
Sep 09 20:42:30.798922 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:30.798894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qk9rl" event={"ID":"cad3822b-6c99-478f-a17a-26ca43f5fd7e","Type":"ContainerStarted","Data":"d69937915bd3739c575b791d12b3c61fd48f301f714fccdc9f9aa7715a50bff7"}
Sep 09 20:42:31.804265 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:31.804229    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qk9rl" event={"ID":"cad3822b-6c99-478f-a17a-26ca43f5fd7e","Type":"ContainerStarted","Data":"f412b8cb27dd1956413e8725dc96c0fe544f56084de2c7646a7e79ba35ba7594"}
Sep 09 20:42:31.819250 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:31.819201    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qk9rl" podStartSLOduration=130.7613758 podStartE2EDuration="2m11.819185725s" totalImagesPullingTime="1.057809925s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:42:31.818545866 +0000 UTC m=+132.010383468" watchObservedRunningTime="2026-09-09 20:42:31.819185725 +0000 UTC m=+132.011023324"
Sep 09 20:42:31.900039 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:31.900007    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:32.000835 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:32.000805    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:35.885916 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:35.885879    2566 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" podUID="78303679-3153-4d46-91c0-dd93139fa25d" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 09 20:42:35.886268 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:35.885941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79"
Sep 09 20:42:35.886416 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:35.886387    2566 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c6f01f664e3e45f4fe3ff5f25a4159b571f18fd18919b25bf8ed66ec0a166e7b"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 09 20:42:35.886460 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:35.886435    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" podUID="78303679-3153-4d46-91c0-dd93139fa25d" containerName="service-proxy" containerID="cri-o://c6f01f664e3e45f4fe3ff5f25a4159b571f18fd18919b25bf8ed66ec0a166e7b" gracePeriod=30
Sep 09 20:42:36.821615 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:36.821578    2566 generic.go:356] "Generic (PLEG): container finished" podID="78303679-3153-4d46-91c0-dd93139fa25d" containerID="c6f01f664e3e45f4fe3ff5f25a4159b571f18fd18919b25bf8ed66ec0a166e7b" exitCode=2
Sep 09 20:42:36.821801 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:36.821623    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" event={"ID":"78303679-3153-4d46-91c0-dd93139fa25d","Type":"ContainerDied","Data":"c6f01f664e3e45f4fe3ff5f25a4159b571f18fd18919b25bf8ed66ec0a166e7b"}
Sep 09 20:42:36.821801 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:36.821665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-847c5d47b4-9td79" event={"ID":"78303679-3153-4d46-91c0-dd93139fa25d","Type":"ContainerStarted","Data":"2f29e80ccf56e852e198f3ea393d425db709464975ab45f31556c7654e0141d1"}
Sep 09 20:42:37.230519 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:37.230486    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:37.296029 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:37.295995    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:38.345813 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.345780    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9" podUID="a8752ff6-9b1b-4b50-8ece-1f045b1393f4" containerName="registry" containerID="cri-o://7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6" gracePeriod=30
Sep 09 20:42:38.581430 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.581405    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:42:38.645194 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645118    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") pod \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") "
Sep 09 20:42:38.645194 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-certificates\") pod \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") "
Sep 09 20:42:38.645194 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645183    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-image-registry-private-configuration\") pod \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") "
Sep 09 20:42:38.645464 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645203    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cplnn\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-kube-api-access-cplnn\") pod \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") "
Sep 09 20:42:38.645464 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645274    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-bound-sa-token\") pod \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") "
Sep 09 20:42:38.645464 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645372    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-ca-trust-extracted\") pod \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") "
Sep 09 20:42:38.645464 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645410    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-installation-pull-secrets\") pod \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") "
Sep 09 20:42:38.645464 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645437    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-trusted-ca\") pod \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\" (UID: \"a8752ff6-9b1b-4b50-8ece-1f045b1393f4\") "
Sep 09 20:42:38.645709 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645680    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "a8752ff6-9b1b-4b50-8ece-1f045b1393f4" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:42:38.646053 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.645988    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "a8752ff6-9b1b-4b50-8ece-1f045b1393f4" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:42:38.647658 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.647617    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "a8752ff6-9b1b-4b50-8ece-1f045b1393f4" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:42:38.647767 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.647725    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "a8752ff6-9b1b-4b50-8ece-1f045b1393f4" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:42:38.647910 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.647883    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "a8752ff6-9b1b-4b50-8ece-1f045b1393f4" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:42:38.648016 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.647993    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "a8752ff6-9b1b-4b50-8ece-1f045b1393f4" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:42:38.648099 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.648080    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-kube-api-access-cplnn" (OuterVolumeSpecName: "kube-api-access-cplnn") pod "a8752ff6-9b1b-4b50-8ece-1f045b1393f4" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4"). InnerVolumeSpecName "kube-api-access-cplnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:42:38.653990 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.653966    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "a8752ff6-9b1b-4b50-8ece-1f045b1393f4" (UID: "a8752ff6-9b1b-4b50-8ece-1f045b1393f4"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:42:38.745992 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.745970    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-tls\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:42:38.745992 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.745990    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-registry-certificates\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:42:38.746097 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.746001    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-image-registry-private-configuration\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:42:38.746097 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.746011    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cplnn\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-kube-api-access-cplnn\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:42:38.746097 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.746019    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-bound-sa-token\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:42:38.746097 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.746026    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-ca-trust-extracted\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:42:38.746097 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.746034    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-installation-pull-secrets\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:42:38.746097 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.746042    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a8752ff6-9b1b-4b50-8ece-1f045b1393f4-trusted-ca\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:42:38.754988 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.754956    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:38.830300 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.830275    2566 generic.go:356] "Generic (PLEG): container finished" podID="a8752ff6-9b1b-4b50-8ece-1f045b1393f4" containerID="7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6" exitCode=0
Sep 09 20:42:38.830423 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.830332    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9"
Sep 09 20:42:38.830423 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.830337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9" event={"ID":"a8752ff6-9b1b-4b50-8ece-1f045b1393f4","Type":"ContainerDied","Data":"7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6"}
Sep 09 20:42:38.830423 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.830418    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-cb854fdc9-m6sx9" event={"ID":"a8752ff6-9b1b-4b50-8ece-1f045b1393f4","Type":"ContainerDied","Data":"ced81d8d7d88fa1da22639b0b93c6cb81a81794ffdb840c3cd89dfa02b196a2d"}
Sep 09 20:42:38.830578 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.830448    2566 scope.go:117] "RemoveContainer" containerID="7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6"
Sep 09 20:42:38.838112 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.838094    2566 scope.go:117] "RemoveContainer" containerID="7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6"
Sep 09 20:42:38.838399 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:42:38.838377    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6\": container with ID starting with 7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6 not found: ID does not exist" containerID="7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6"
Sep 09 20:42:38.838468 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.838406    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6"} err="failed to get container status \"7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6\": rpc error: code = NotFound desc = could not find container \"7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6\": container with ID starting with 7236f2ec52aa813fd074ce57431c20f98fdb939313df7d1c609ce456564b26b6 not found: ID does not exist"
Sep 09 20:42:38.852934 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.852910    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-cb854fdc9-m6sx9"]
Sep 09 20:42:38.857480 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:38.857461    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-cb854fdc9-m6sx9"]
Sep 09 20:42:40.284033 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:40.284000    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8752ff6-9b1b-4b50-8ece-1f045b1393f4" path="/var/lib/kubelet/pods/a8752ff6-9b1b-4b50-8ece-1f045b1393f4/volumes"
Sep 09 20:42:52.224961 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:52.224920    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:52.297450 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:52.297407    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:57.597259 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:57.597231    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rxdcl_e562a150-9463-4b3d-a8be-a04e8703a450/serve-healthcheck-canary/0.log"
Sep 09 20:42:58.798329 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:58.798283    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:58.898260 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:58.898224    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:42:58.966428 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:42:58.966402    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:25.755757 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:25.755723    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:25.948661 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:25.948626    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:34.465781 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:34.465745    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:34.521808 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:34.521779    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:34.629024 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:34.628989    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:35.211411 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:35.211379    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:35.265868 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:35.265843    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:40.977642 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:40.977604    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:41.029004 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:41.028965    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:52.902527 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:52.902486    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:52.956815 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:52.956777    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:56.561559 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:56.561518    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:56.618514 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:56.618485    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:56.673500 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:56.673471    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:57.886886 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:57.886844    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:57.987872 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:57.987840    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:43:58.054032 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:43:58.053999    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:44:27.663956 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:44:27.663911    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:08.535424 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:08.535346    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:08.640707 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:08.640675    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:20.230727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:20.230698    2566 kubelet.go:1648] "Image garbage collection succeeded"
Sep 09 20:45:25.880507 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:25.880474    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:35.957124 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:35.957087    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:37.036031 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:37.035996    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:40.138080 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:40.138045    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:40.236444 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:40.236415    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:41.717769 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:41.717729    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:42.345672 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:42.345629    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:42.461376 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:42.461343    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:44.176999 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:44.176961    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:44.778423 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:44.778387    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:47.624037 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:47.623999    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:47.823387 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:47.823357    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:53.560670 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:53.560629    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:55.206789 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.206757    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sqdxh"]
Sep 09 20:45:55.207152 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.207013    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8752ff6-9b1b-4b50-8ece-1f045b1393f4" containerName="registry"
Sep 09 20:45:55.207152 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.207025    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8752ff6-9b1b-4b50-8ece-1f045b1393f4" containerName="registry"
Sep 09 20:45:55.207152 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.207063    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8752ff6-9b1b-4b50-8ece-1f045b1393f4" containerName="registry"
Sep 09 20:45:55.209660 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.209645    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sqdxh"
Sep 09 20:45:55.215823 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.215802    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 09 20:45:55.215958 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.215938    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 09 20:45:55.216725 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.216707    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ljtp5\""
Sep 09 20:45:55.226884 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.226858    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sqdxh"]
Sep 09 20:45:55.278822 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.278798    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e271960e-eb3b-4afc-b990-486638768d6f-bound-sa-token\") pod \"cert-manager-55b4d954d5-sqdxh\" (UID: \"e271960e-eb3b-4afc-b990-486638768d6f\") " pod="cert-manager/cert-manager-55b4d954d5-sqdxh"
Sep 09 20:45:55.278914 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.278836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smkgr\" (UniqueName: \"kubernetes.io/projected/e271960e-eb3b-4afc-b990-486638768d6f-kube-api-access-smkgr\") pod \"cert-manager-55b4d954d5-sqdxh\" (UID: \"e271960e-eb3b-4afc-b990-486638768d6f\") " pod="cert-manager/cert-manager-55b4d954d5-sqdxh"
Sep 09 20:45:55.379178 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.379158    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smkgr\" (UniqueName: \"kubernetes.io/projected/e271960e-eb3b-4afc-b990-486638768d6f-kube-api-access-smkgr\") pod \"cert-manager-55b4d954d5-sqdxh\" (UID: \"e271960e-eb3b-4afc-b990-486638768d6f\") " pod="cert-manager/cert-manager-55b4d954d5-sqdxh"
Sep 09 20:45:55.379299 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.379224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e271960e-eb3b-4afc-b990-486638768d6f-bound-sa-token\") pod \"cert-manager-55b4d954d5-sqdxh\" (UID: \"e271960e-eb3b-4afc-b990-486638768d6f\") " pod="cert-manager/cert-manager-55b4d954d5-sqdxh"
Sep 09 20:45:55.390341 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.389784    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e271960e-eb3b-4afc-b990-486638768d6f-bound-sa-token\") pod \"cert-manager-55b4d954d5-sqdxh\" (UID: \"e271960e-eb3b-4afc-b990-486638768d6f\") " pod="cert-manager/cert-manager-55b4d954d5-sqdxh"
Sep 09 20:45:55.390341 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.389926    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smkgr\" (UniqueName: \"kubernetes.io/projected/e271960e-eb3b-4afc-b990-486638768d6f-kube-api-access-smkgr\") pod \"cert-manager-55b4d954d5-sqdxh\" (UID: \"e271960e-eb3b-4afc-b990-486638768d6f\") " pod="cert-manager/cert-manager-55b4d954d5-sqdxh"
Sep 09 20:45:55.518658 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.518636    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-sqdxh"
Sep 09 20:45:55.858045 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:45:55.858016    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode271960e_eb3b_4afc_b990_486638768d6f.slice/crio-7998a6f85f0cb3c6cffad1d19f54aed2bf911707245ebd5901252540c366274e WatchSource:0}: Error finding container 7998a6f85f0cb3c6cffad1d19f54aed2bf911707245ebd5901252540c366274e: Status 404 returned error can't find the container with id 7998a6f85f0cb3c6cffad1d19f54aed2bf911707245ebd5901252540c366274e
Sep 09 20:45:55.859839 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.859823    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:45:55.869579 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:55.869555    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-sqdxh"]
Sep 09 20:45:56.050907 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:56.050877    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:56.193508 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:56.193446    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:45:56.441695 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:56.441659    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sqdxh" event={"ID":"e271960e-eb3b-4afc-b990-486638768d6f","Type":"ContainerStarted","Data":"7998a6f85f0cb3c6cffad1d19f54aed2bf911707245ebd5901252540c366274e"}
Sep 09 20:45:59.456093 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:59.456004    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-sqdxh" event={"ID":"e271960e-eb3b-4afc-b990-486638768d6f","Type":"ContainerStarted","Data":"6f10b3227f85aa243b6834d8e399b22552446c535052d35a7b469ef686d647e1"}
Sep 09 20:45:59.475089 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:45:59.475046    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-sqdxh" podStartSLOduration=1.267810025 podStartE2EDuration="4.475032192s" totalImagesPullingTime="3.207222167s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:45:59.47492782 +0000 UTC m=+339.666765419" watchObservedRunningTime="2026-09-09 20:45:59.475032192 +0000 UTC m=+339.666869791"
Sep 09 20:46:00.908637 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:00.908602    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:01.121276 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:01.121239    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:02.947738 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:02.947702    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:05.208404 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:05.208363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:05.358670 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:05.358632    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:17.449533 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:17.449486    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:17.672877 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:17.672842    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:17.877618 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:17.877585    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:18.635715 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.635689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:18.737345 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.737299    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:18.789057 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.789028    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"]
Sep 09 20:46:18.792285 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.792269    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.795126 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.795108    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 09 20:46:18.795222 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.795123    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 09 20:46:18.795398 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.795382    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-4jzzz\""
Sep 09 20:46:18.795480 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.795395    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 09 20:46:18.795613 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.795598    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 09 20:46:18.812800 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.812776    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"]
Sep 09 20:46:18.841932 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.841907    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d454cced-f2f7-47c8-bb0a-6fb136622c54-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.842022 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.841942    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d454cced-f2f7-47c8-bb0a-6fb136622c54-manifests\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.842022 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.841974    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d454cced-f2f7-47c8-bb0a-6fb136622c54-tmp\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.842022 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.841998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d454cced-f2f7-47c8-bb0a-6fb136622c54-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.842022 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.842015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc8bc\" (UniqueName: \"kubernetes.io/projected/d454cced-f2f7-47c8-bb0a-6fb136622c54-kube-api-access-cc8bc\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.942399 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.942325    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d454cced-f2f7-47c8-bb0a-6fb136622c54-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.942399 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.942361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d454cced-f2f7-47c8-bb0a-6fb136622c54-manifests\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.942399 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.942386    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d454cced-f2f7-47c8-bb0a-6fb136622c54-tmp\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.942578 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.942415    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d454cced-f2f7-47c8-bb0a-6fb136622c54-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.942578 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.942541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc8bc\" (UniqueName: \"kubernetes.io/projected/d454cced-f2f7-47c8-bb0a-6fb136622c54-kube-api-access-cc8bc\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.942768 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.942747    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/d454cced-f2f7-47c8-bb0a-6fb136622c54-manifests\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.944711 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.944682    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d454cced-f2f7-47c8-bb0a-6fb136622c54-tmp\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.944902 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.944885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/d454cced-f2f7-47c8-bb0a-6fb136622c54-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.945005 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.944987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d454cced-f2f7-47c8-bb0a-6fb136622c54-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.958487 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.958470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc8bc\" (UniqueName: \"kubernetes.io/projected/d454cced-f2f7-47c8-bb0a-6fb136622c54-kube-api-access-cc8bc\") pod \"opendatahub-operator-controller-manager-6d46659dfc-tz2zw\" (UID: \"d454cced-f2f7-47c8-bb0a-6fb136622c54\") " pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:18.978462 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:18.978435    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:19.074326 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:19.074279    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:19.102202 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:19.102181    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:19.177936 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:19.177906    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:19.278836 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:19.278808    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:19.423749 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:19.423722    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"]
Sep 09 20:46:19.472969 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:46:19.472940    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd454cced_f2f7_47c8_bb0a_6fb136622c54.slice/crio-194c831e70f4e67787b603d292a5cb03e03a4db5bda407fdfe3f05f8bc442071 WatchSource:0}: Error finding container 194c831e70f4e67787b603d292a5cb03e03a4db5bda407fdfe3f05f8bc442071: Status 404 returned error can't find the container with id 194c831e70f4e67787b603d292a5cb03e03a4db5bda407fdfe3f05f8bc442071
Sep 09 20:46:19.519036 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:19.519008    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw" event={"ID":"d454cced-f2f7-47c8-bb0a-6fb136622c54","Type":"ContainerStarted","Data":"194c831e70f4e67787b603d292a5cb03e03a4db5bda407fdfe3f05f8bc442071"}
Sep 09 20:46:19.528084 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:19.528057    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:23.534939 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:23.534900    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw" event={"ID":"d454cced-f2f7-47c8-bb0a-6fb136622c54","Type":"ContainerStarted","Data":"b4f6ff352812112bfb642878546cf232680e8ba30615d33b16ddb8b808d4f8ae"}
Sep 09 20:46:24.539612 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:24.539573    2566 generic.go:356] "Generic (PLEG): container finished" podID="d454cced-f2f7-47c8-bb0a-6fb136622c54" containerID="b4f6ff352812112bfb642878546cf232680e8ba30615d33b16ddb8b808d4f8ae" exitCode=0
Sep 09 20:46:24.540090 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:24.539625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw" event={"ID":"d454cced-f2f7-47c8-bb0a-6fb136622c54","Type":"ContainerDied","Data":"b4f6ff352812112bfb642878546cf232680e8ba30615d33b16ddb8b808d4f8ae"}
Sep 09 20:46:25.544453 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:25.544417    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw" event={"ID":"d454cced-f2f7-47c8-bb0a-6fb136622c54","Type":"ContainerStarted","Data":"09f45d2cf7063f7da4e26cbe2c244d9c4b8d3ff9ee383c9d9565484a90098712"}
Sep 09 20:46:25.544905 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:25.544559    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:25.621224 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:25.621177    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw" podStartSLOduration=3.092557283 podStartE2EDuration="7.621161985s" totalImagesPullingTime="3.462891258s" totalInitContainerRuntime="1.065713444s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:18 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:25.621150837 +0000 UTC m=+365.812988436" watchObservedRunningTime="2026-09-09 20:46:25.621161985 +0000 UTC m=+365.812999581"
Sep 09 20:46:33.033543 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:33.033506    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:33.254854 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:33.254811    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:36.550476 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:36.550442    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d46659dfc-tz2zw"
Sep 09 20:46:39.196986 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:39.196952    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:39.466818 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:39.466734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:45.403598 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.403555    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:45.451649 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.451625    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"]
Sep 09 20:46:45.454692 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.454563    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.459722 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.459702    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-gxkjq\""
Sep 09 20:46:45.459828 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.459775    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 09 20:46:45.460059 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.460041    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 09 20:46:45.460107 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.460077    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 09 20:46:45.460283 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.460269    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 09 20:46:45.479985 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.479961    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"]
Sep 09 20:46:45.519372 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.519345    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dps2g\" (UniqueName: \"kubernetes.io/projected/a175b88d-bb68-48d0-ad85-268cf209de6d-kube-api-access-dps2g\") pod \"kube-auth-proxy-6c8d5d5c67-mtqtw\" (UID: \"a175b88d-bb68-48d0-ad85-268cf209de6d\") " pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.519465 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.519412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a175b88d-bb68-48d0-ad85-268cf209de6d-tmp\") pod \"kube-auth-proxy-6c8d5d5c67-mtqtw\" (UID: \"a175b88d-bb68-48d0-ad85-268cf209de6d\") " pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.519465 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.519436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175b88d-bb68-48d0-ad85-268cf209de6d-tls-certs\") pod \"kube-auth-proxy-6c8d5d5c67-mtqtw\" (UID: \"a175b88d-bb68-48d0-ad85-268cf209de6d\") " pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.617899 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.617873    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:45.620757 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.620737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a175b88d-bb68-48d0-ad85-268cf209de6d-tmp\") pod \"kube-auth-proxy-6c8d5d5c67-mtqtw\" (UID: \"a175b88d-bb68-48d0-ad85-268cf209de6d\") " pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.620839 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.620770    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175b88d-bb68-48d0-ad85-268cf209de6d-tls-certs\") pod \"kube-auth-proxy-6c8d5d5c67-mtqtw\" (UID: \"a175b88d-bb68-48d0-ad85-268cf209de6d\") " pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.620839 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.620816    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps2g\" (UniqueName: \"kubernetes.io/projected/a175b88d-bb68-48d0-ad85-268cf209de6d-kube-api-access-dps2g\") pod \"kube-auth-proxy-6c8d5d5c67-mtqtw\" (UID: \"a175b88d-bb68-48d0-ad85-268cf209de6d\") " pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.622911 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.622887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a175b88d-bb68-48d0-ad85-268cf209de6d-tmp\") pod \"kube-auth-proxy-6c8d5d5c67-mtqtw\" (UID: \"a175b88d-bb68-48d0-ad85-268cf209de6d\") " pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.623049 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.623033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a175b88d-bb68-48d0-ad85-268cf209de6d-tls-certs\") pod \"kube-auth-proxy-6c8d5d5c67-mtqtw\" (UID: \"a175b88d-bb68-48d0-ad85-268cf209de6d\") " pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.632530 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.632509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dps2g\" (UniqueName: \"kubernetes.io/projected/a175b88d-bb68-48d0-ad85-268cf209de6d-kube-api-access-dps2g\") pod \"kube-auth-proxy-6c8d5d5c67-mtqtw\" (UID: \"a175b88d-bb68-48d0-ad85-268cf209de6d\") " pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:45.763760 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:45.763738    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"
Sep 09 20:46:46.046771 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:46.046689    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw"]
Sep 09 20:46:46.050155 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:46:46.050129    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda175b88d_bb68_48d0_ad85_268cf209de6d.slice/crio-063aabd035109365aa810caf52e7c949239a2b07d20dbad258ea40e3aa2ad07f WatchSource:0}: Error finding container 063aabd035109365aa810caf52e7c949239a2b07d20dbad258ea40e3aa2ad07f: Status 404 returned error can't find the container with id 063aabd035109365aa810caf52e7c949239a2b07d20dbad258ea40e3aa2ad07f
Sep 09 20:46:46.613782 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:46.613751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw" event={"ID":"a175b88d-bb68-48d0-ad85-268cf209de6d","Type":"ContainerStarted","Data":"063aabd035109365aa810caf52e7c949239a2b07d20dbad258ea40e3aa2ad07f"}
Sep 09 20:46:48.429899 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.429855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:48.463959 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.463931    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"]
Sep 09 20:46:48.467201 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.467182    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.482699 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.482681    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 09 20:46:48.482824 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.482801    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 09 20:46:48.501082 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.501056    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 09 20:46:48.501200 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.501157    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:46:48.505281 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.505261    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-s9d4v\""
Sep 09 20:46:48.544235 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.544204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.544381 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.544268    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828v5\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-kube-api-access-828v5\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.544381 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.544337    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.544512 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.544418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.544512 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.544451    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/966b5679-e9bd-41b0-8f93-2404e5be4417-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.544512 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.544490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.544649 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.544519    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.549803 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.549777    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"]
Sep 09 20:46:48.645024 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.644988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.645185 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.645037    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/966b5679-e9bd-41b0-8f93-2404e5be4417-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.645185 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.645070    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.645185 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.645091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.645391 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.645276    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.645391 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.645302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828v5\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-kube-api-access-828v5\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.645391 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.645347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.646097 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.646060    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.647747 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.647721    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/966b5679-e9bd-41b0-8f93-2404e5be4417-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.647860 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.647732    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.647924 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.647896    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.647997 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.647970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.663523 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.663502    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.663627 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.663557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-828v5\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-kube-api-access-828v5\") pod \"istiod-openshift-gateway-7b7995d5cd-swzr7\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:48.698867 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.698809    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:48.777709 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:48.777681    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:49.138069 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:49.138045    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"]
Sep 09 20:46:49.140002 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:46:49.139974    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966b5679_e9bd_41b0_8f93_2404e5be4417.slice/crio-e501b911b54c74f07ec4d286f2f69e4770d6635a49478702c532dac5afbdf541 WatchSource:0}: Error finding container e501b911b54c74f07ec4d286f2f69e4770d6635a49478702c532dac5afbdf541: Status 404 returned error can't find the container with id e501b911b54c74f07ec4d286f2f69e4770d6635a49478702c532dac5afbdf541
Sep 09 20:46:49.626663 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:49.626625    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7" event={"ID":"966b5679-e9bd-41b0-8f93-2404e5be4417","Type":"ContainerStarted","Data":"e501b911b54c74f07ec4d286f2f69e4770d6635a49478702c532dac5afbdf541"}
Sep 09 20:46:49.628074 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:49.628050    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw" event={"ID":"a175b88d-bb68-48d0-ad85-268cf209de6d","Type":"ContainerStarted","Data":"54a96c405fe859a68f57ddfcc2b55cbc6d23b886918850c2ad179775bc058475"}
Sep 09 20:46:49.676610 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:49.676568    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6c8d5d5c67-mtqtw" podStartSLOduration=1.816739467 podStartE2EDuration="4.676553525s" totalImagesPullingTime="2.859814058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:49.676087355 +0000 UTC m=+389.867924955" watchObservedRunningTime="2026-09-09 20:46:49.676553525 +0000 UTC m=+389.868391124"
Sep 09 20:46:50.423693 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:50.423662    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:50.645762 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:50.645726    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:50.821743 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:50.821690    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:51.404698 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:51.404660    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:51.920460 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:51.920430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:51.996911 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:51.996879    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:46:51.996992 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:51.996938    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:46:52.244080 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:52.244046    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:52.640886 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:52.640805    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7" event={"ID":"966b5679-e9bd-41b0-8f93-2404e5be4417","Type":"ContainerStarted","Data":"ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080"}
Sep 09 20:46:52.640886 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:52.640877    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:53.077405 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:53.077361    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:53.325830 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:53.325794    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:53.645733 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:53.645705    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-swzr7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 20:46:53.645884 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:53.645780    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7" podUID="966b5679-e9bd-41b0-8f93-2404e5be4417" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 20:46:54.416264 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:54.416227    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:54.744789 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:54.744755    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:54.994616 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:54.994576    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:56.644835 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:56.644802    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:46:56.705861 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:56.705815    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7" podStartSLOduration=5.851098572 podStartE2EDuration="8.705800188s" totalImagesPullingTime="2.854701616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:46:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:46:52.680238441 +0000 UTC m=+392.872076041" watchObservedRunningTime="2026-09-09 20:46:56.705800188 +0000 UTC m=+396.897637786"
Sep 09 20:46:59.300452 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:59.300408    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:46:59.689070 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:46:59.688996    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:08.008877 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.008836    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"]
Sep 09 20:47:08.015413 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.015387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:08.020265 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.020238    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 09 20:47:08.020411 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.020269    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-6bpm2\""
Sep 09 20:47:08.061720 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.061694    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:08.065519 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.065494    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"]
Sep 09 20:47:08.067406 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.067391    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:08.072649 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.072629    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 09 20:47:08.073106 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.073087    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-w5pxt\""
Sep 09 20:47:08.185628 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.185594    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3eb13ab3-9716-48f0-adfe-be6b20e57029-cert\") pod \"kserve-controller-manager-d6f4b65b8-k79s4\" (UID: \"3eb13ab3-9716-48f0-adfe-be6b20e57029\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:08.185628 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.185627    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d70471c-1528-4fbc-ad80-10f52b4dda5e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vfgww\" (UID: \"8d70471c-1528-4fbc-ad80-10f52b4dda5e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:08.185811 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.185657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qshsh\" (UniqueName: \"kubernetes.io/projected/8d70471c-1528-4fbc-ad80-10f52b4dda5e-kube-api-access-qshsh\") pod \"llmisvc-controller-manager-777c6bd7d5-vfgww\" (UID: \"8d70471c-1528-4fbc-ad80-10f52b4dda5e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:08.185811 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.185732    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rkpp\" (UniqueName: \"kubernetes.io/projected/3eb13ab3-9716-48f0-adfe-be6b20e57029-kube-api-access-6rkpp\") pod \"kserve-controller-manager-d6f4b65b8-k79s4\" (UID: \"3eb13ab3-9716-48f0-adfe-be6b20e57029\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:08.200763 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.200740    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"]
Sep 09 20:47:08.230465 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.230442    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"]
Sep 09 20:47:08.286955 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.286894    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3eb13ab3-9716-48f0-adfe-be6b20e57029-cert\") pod \"kserve-controller-manager-d6f4b65b8-k79s4\" (UID: \"3eb13ab3-9716-48f0-adfe-be6b20e57029\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:08.286955 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.286923    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d70471c-1528-4fbc-ad80-10f52b4dda5e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vfgww\" (UID: \"8d70471c-1528-4fbc-ad80-10f52b4dda5e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:08.286955 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.286944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qshsh\" (UniqueName: \"kubernetes.io/projected/8d70471c-1528-4fbc-ad80-10f52b4dda5e-kube-api-access-qshsh\") pod \"llmisvc-controller-manager-777c6bd7d5-vfgww\" (UID: \"8d70471c-1528-4fbc-ad80-10f52b4dda5e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:08.287167 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.286969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rkpp\" (UniqueName: \"kubernetes.io/projected/3eb13ab3-9716-48f0-adfe-be6b20e57029-kube-api-access-6rkpp\") pod \"kserve-controller-manager-d6f4b65b8-k79s4\" (UID: \"3eb13ab3-9716-48f0-adfe-be6b20e57029\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:08.287167 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:47:08.287047    2566 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Sep 09 20:47:08.287167 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:47:08.287121    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3eb13ab3-9716-48f0-adfe-be6b20e57029-cert podName:3eb13ab3-9716-48f0-adfe-be6b20e57029 nodeName:}" failed. No retries permitted until 2026-09-09 20:47:08.787101562 +0000 UTC m=+408.978939155 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/3eb13ab3-9716-48f0-adfe-be6b20e57029-cert") pod "kserve-controller-manager-d6f4b65b8-k79s4" (UID: "3eb13ab3-9716-48f0-adfe-be6b20e57029") : secret "kserve-webhook-server-cert" not found
Sep 09 20:47:08.289327 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.289290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8d70471c-1528-4fbc-ad80-10f52b4dda5e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-vfgww\" (UID: \"8d70471c-1528-4fbc-ad80-10f52b4dda5e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:08.310766 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.310742    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rkpp\" (UniqueName: \"kubernetes.io/projected/3eb13ab3-9716-48f0-adfe-be6b20e57029-kube-api-access-6rkpp\") pod \"kserve-controller-manager-d6f4b65b8-k79s4\" (UID: \"3eb13ab3-9716-48f0-adfe-be6b20e57029\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:08.313865 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.313840    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qshsh\" (UniqueName: \"kubernetes.io/projected/8d70471c-1528-4fbc-ad80-10f52b4dda5e-kube-api-access-qshsh\") pod \"llmisvc-controller-manager-777c6bd7d5-vfgww\" (UID: \"8d70471c-1528-4fbc-ad80-10f52b4dda5e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:08.376823 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.376798    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:08.442221 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.442190    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:08.691498 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.691422    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"]
Sep 09 20:47:08.695191 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:47:08.695162    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8d70471c_1528_4fbc_ad80_10f52b4dda5e.slice/crio-ad2fc3b7f677133d42e5884c063a5c5aa08e5caef79c0e9860de3884a7277ccb WatchSource:0}: Error finding container ad2fc3b7f677133d42e5884c063a5c5aa08e5caef79c0e9860de3884a7277ccb: Status 404 returned error can't find the container with id ad2fc3b7f677133d42e5884c063a5c5aa08e5caef79c0e9860de3884a7277ccb
Sep 09 20:47:08.790864 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.790838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3eb13ab3-9716-48f0-adfe-be6b20e57029-cert\") pod \"kserve-controller-manager-d6f4b65b8-k79s4\" (UID: \"3eb13ab3-9716-48f0-adfe-be6b20e57029\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:08.792995 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.792978    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3eb13ab3-9716-48f0-adfe-be6b20e57029-cert\") pod \"kserve-controller-manager-d6f4b65b8-k79s4\" (UID: \"3eb13ab3-9716-48f0-adfe-be6b20e57029\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:08.894593 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.894568    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:08.924596 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:08.924574    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:09.222867 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:09.222824    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"]
Sep 09 20:47:09.225257 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:47:09.225236    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eb13ab3_9716_48f0_adfe_be6b20e57029.slice/crio-32f19ef08d0c44af8eefaed52c63abea0298d77e310ed1fac30ea0fbab42116e WatchSource:0}: Error finding container 32f19ef08d0c44af8eefaed52c63abea0298d77e310ed1fac30ea0fbab42116e: Status 404 returned error can't find the container with id 32f19ef08d0c44af8eefaed52c63abea0298d77e310ed1fac30ea0fbab42116e
Sep 09 20:47:09.487827 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:09.487797    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:09.701174 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:09.701134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4" event={"ID":"3eb13ab3-9716-48f0-adfe-be6b20e57029","Type":"ContainerStarted","Data":"32f19ef08d0c44af8eefaed52c63abea0298d77e310ed1fac30ea0fbab42116e"}
Sep 09 20:47:09.703053 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:09.703013    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww" event={"ID":"8d70471c-1528-4fbc-ad80-10f52b4dda5e","Type":"ContainerStarted","Data":"ad2fc3b7f677133d42e5884c063a5c5aa08e5caef79c0e9860de3884a7277ccb"}
Sep 09 20:47:09.740539 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:09.740463    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:09.775635 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:09.775574    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"]
Sep 09 20:47:09.775934 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:09.775806    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7" podUID="966b5679-e9bd-41b0-8f93-2404e5be4417" containerName="discovery" containerID="cri-o://ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080" gracePeriod=30
Sep 09 20:47:10.043147 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.043108    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:47:10.203340 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.203295    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-kubeconfig\") pod \"966b5679-e9bd-41b0-8f93-2404e5be4417\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") "
Sep 09 20:47:10.203515 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.203358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-828v5\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-kube-api-access-828v5\") pod \"966b5679-e9bd-41b0-8f93-2404e5be4417\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") "
Sep 09 20:47:10.203515 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.203390    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-token\") pod \"966b5679-e9bd-41b0-8f93-2404e5be4417\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") "
Sep 09 20:47:10.203515 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.203436    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-cacerts\") pod \"966b5679-e9bd-41b0-8f93-2404e5be4417\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") "
Sep 09 20:47:10.203515 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.203470    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-dns-cert\") pod \"966b5679-e9bd-41b0-8f93-2404e5be4417\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") "
Sep 09 20:47:10.203515 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.203498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-ca-configmap\") pod \"966b5679-e9bd-41b0-8f93-2404e5be4417\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") "
Sep 09 20:47:10.203763 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.203526    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/966b5679-e9bd-41b0-8f93-2404e5be4417-local-certs\") pod \"966b5679-e9bd-41b0-8f93-2404e5be4417\" (UID: \"966b5679-e9bd-41b0-8f93-2404e5be4417\") "
Sep 09 20:47:10.204595 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.204530    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "966b5679-e9bd-41b0-8f93-2404e5be4417" (UID: "966b5679-e9bd-41b0-8f93-2404e5be4417"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:47:10.206490 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.206458    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-cacerts" (OuterVolumeSpecName: "cacerts") pod "966b5679-e9bd-41b0-8f93-2404e5be4417" (UID: "966b5679-e9bd-41b0-8f93-2404e5be4417"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:10.206606 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.206570    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-kube-api-access-828v5" (OuterVolumeSpecName: "kube-api-access-828v5") pod "966b5679-e9bd-41b0-8f93-2404e5be4417" (UID: "966b5679-e9bd-41b0-8f93-2404e5be4417"). InnerVolumeSpecName "kube-api-access-828v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:10.206754 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.206727    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "966b5679-e9bd-41b0-8f93-2404e5be4417" (UID: "966b5679-e9bd-41b0-8f93-2404e5be4417"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:10.206834 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.206775    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-token" (OuterVolumeSpecName: "istio-token") pod "966b5679-e9bd-41b0-8f93-2404e5be4417" (UID: "966b5679-e9bd-41b0-8f93-2404e5be4417"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:47:10.207220 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.207204    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "966b5679-e9bd-41b0-8f93-2404e5be4417" (UID: "966b5679-e9bd-41b0-8f93-2404e5be4417"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:47:10.207772 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.207713    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/966b5679-e9bd-41b0-8f93-2404e5be4417-local-certs" (OuterVolumeSpecName: "local-certs") pod "966b5679-e9bd-41b0-8f93-2404e5be4417" (UID: "966b5679-e9bd-41b0-8f93-2404e5be4417"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:47:10.305025 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.304999    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-dns-cert\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:47:10.305344 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.305028    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-csr-ca-configmap\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:47:10.305344 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.305043    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/966b5679-e9bd-41b0-8f93-2404e5be4417-local-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:47:10.305344 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.305052    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-kubeconfig\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:47:10.305344 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.305059    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-828v5\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-kube-api-access-828v5\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:47:10.305344 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.305067    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/966b5679-e9bd-41b0-8f93-2404e5be4417-istio-token\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:47:10.305344 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.305075    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/966b5679-e9bd-41b0-8f93-2404e5be4417-cacerts\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:47:10.709253 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.709173    2566 generic.go:356] "Generic (PLEG): container finished" podID="966b5679-e9bd-41b0-8f93-2404e5be4417" containerID="ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080" exitCode=0
Sep 09 20:47:10.709253 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.709226    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"
Sep 09 20:47:10.709253 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.709236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7" event={"ID":"966b5679-e9bd-41b0-8f93-2404e5be4417","Type":"ContainerDied","Data":"ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080"}
Sep 09 20:47:10.709486 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.709324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7" event={"ID":"966b5679-e9bd-41b0-8f93-2404e5be4417","Type":"ContainerDied","Data":"e501b911b54c74f07ec4d286f2f69e4770d6635a49478702c532dac5afbdf541"}
Sep 09 20:47:10.709486 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.709349    2566 scope.go:117] "RemoveContainer" containerID="ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080"
Sep 09 20:47:10.716883 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.716865    2566 scope.go:117] "RemoveContainer" containerID="ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080"
Sep 09 20:47:10.717179 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:47:10.717160    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080\": container with ID starting with ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080 not found: ID does not exist" containerID="ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080"
Sep 09 20:47:10.717242 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.717191    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080"} err="failed to get container status \"ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080\": rpc error: code = NotFound desc = could not find container \"ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080\": container with ID starting with ddc745f3d080f3ae44da3a57fd4205942f3bf2fb527cac487f01eaa1694a5080 not found: ID does not exist"
Sep 09 20:47:10.752375 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.752343    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"]
Sep 09 20:47:10.754405 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.754376    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-swzr7"]
Sep 09 20:47:10.865195 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:10.865147    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:11.156380 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:11.156342    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:11.488605 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:11.488564    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:12.284469 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:12.284440    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="966b5679-e9bd-41b0-8f93-2404e5be4417" path="/var/lib/kubelet/pods/966b5679-e9bd-41b0-8f93-2404e5be4417/volumes"
Sep 09 20:47:12.719371 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:12.719277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4" event={"ID":"3eb13ab3-9716-48f0-adfe-be6b20e57029","Type":"ContainerStarted","Data":"375ef119fc185ca1cbaf36f0ea1348d1bd6713eed37a29951e37b29a14f740fe"}
Sep 09 20:47:12.719371 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:12.719365    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:12.720797 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:12.720778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww" event={"ID":"8d70471c-1528-4fbc-ad80-10f52b4dda5e","Type":"ContainerStarted","Data":"539e7b795ad776f47d56baf7d41668e8087c3651d0a700874cc0d51c1f905df7"}
Sep 09 20:47:12.720897 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:12.720821    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:12.827434 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:12.827381    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4" podStartSLOduration=3.097988897 podStartE2EDuration="5.827366285s" totalImagesPullingTime="2.729377388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:12.826680835 +0000 UTC m=+413.018518435" watchObservedRunningTime="2026-09-09 20:47:12.827366285 +0000 UTC m=+413.019203884"
Sep 09 20:47:12.979539 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:12.979490    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww" podStartSLOduration=2.7214333870000003 podStartE2EDuration="5.97947039s" totalImagesPullingTime="3.258037003s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:47:12.965291279 +0000 UTC m=+413.157128879" watchObservedRunningTime="2026-09-09 20:47:12.97947039 +0000 UTC m=+413.171307988"
Sep 09 20:47:29.458185 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:29.458144    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:29.734104 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:29.734073    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:31.966123 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:31.966097    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:32.086800 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:32.086768    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:43.726681 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:43.726605    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-vfgww"
Sep 09 20:47:43.729529 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:43.729504    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-k79s4"
Sep 09 20:47:51.334656 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:51.334614    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:47:51.620003 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:47:51.619913    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:14.934244 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:14.934206    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:17.309479 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.309446    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"]
Sep 09 20:48:17.309823 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.309714    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="966b5679-e9bd-41b0-8f93-2404e5be4417" containerName="discovery"
Sep 09 20:48:17.309823 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.309724    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="966b5679-e9bd-41b0-8f93-2404e5be4417" containerName="discovery"
Sep 09 20:48:17.309823 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.309767    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="966b5679-e9bd-41b0-8f93-2404e5be4417" containerName="discovery"
Sep 09 20:48:17.312503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.312488    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:17.325834 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.325817    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kbdl4\""
Sep 09 20:48:17.326419 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.326404    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 09 20:48:17.330343 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.330329    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 09 20:48:17.339816 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.339794    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"]
Sep 09 20:48:17.353582 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.353561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/5170e3d5-cb80-4c5d-b2d8-cad4f616c46f-kube-api-access-mb66c\") pod \"limitador-operator-controller-manager-74495c69f8-7f85j\" (UID: \"5170e3d5-cb80-4c5d-b2d8-cad4f616c46f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:17.454262 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.454238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/5170e3d5-cb80-4c5d-b2d8-cad4f616c46f-kube-api-access-mb66c\") pod \"limitador-operator-controller-manager-74495c69f8-7f85j\" (UID: \"5170e3d5-cb80-4c5d-b2d8-cad4f616c46f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:17.485124 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.485085    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/5170e3d5-cb80-4c5d-b2d8-cad4f616c46f-kube-api-access-mb66c\") pod \"limitador-operator-controller-manager-74495c69f8-7f85j\" (UID: \"5170e3d5-cb80-4c5d-b2d8-cad4f616c46f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:17.621473 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.621413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:17.990000 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:17.989970    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"]
Sep 09 20:48:17.993734 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:48:17.993708    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5170e3d5_cb80_4c5d_b2d8_cad4f616c46f.slice/crio-08e5c5ff555a8a8efa653edd234caa71b78d25b2d9e7426d3e39813ef03c5da5 WatchSource:0}: Error finding container 08e5c5ff555a8a8efa653edd234caa71b78d25b2d9e7426d3e39813ef03c5da5: Status 404 returned error can't find the container with id 08e5c5ff555a8a8efa653edd234caa71b78d25b2d9e7426d3e39813ef03c5da5
Sep 09 20:48:18.275443 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:18.275363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:18.701532 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:18.701457    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:18.950976 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:18.950946    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j" event={"ID":"5170e3d5-cb80-4c5d-b2d8-cad4f616c46f","Type":"ContainerStarted","Data":"08e5c5ff555a8a8efa653edd234caa71b78d25b2d9e7426d3e39813ef03c5da5"}
Sep 09 20:48:18.986714 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:18.986684    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:19.712709 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:19.712673    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:20.702670 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:20.702640    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:20.959882 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:20.959808    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j" event={"ID":"5170e3d5-cb80-4c5d-b2d8-cad4f616c46f","Type":"ContainerStarted","Data":"cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf"}
Sep 09 20:48:20.959882 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:20.959872    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:21.025076 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:21.025021    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j" podStartSLOduration=1.331187073 podStartE2EDuration="4.025005133s" totalImagesPullingTime="2.69381806s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:21.022763345 +0000 UTC m=+481.214600944" watchObservedRunningTime="2026-09-09 20:48:21.025005133 +0000 UTC m=+481.216842732"
Sep 09 20:48:22.473335 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.473290    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"]
Sep 09 20:48:22.476502 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.476487    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:22.479784 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:22.479762    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:22.479837 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:22.479808    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"kuadrant-operator-controller-manager-dockercfg-4gg4f\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4gg4f\"" type="*v1.Secret"
Sep 09 20:48:22.513232 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.513206    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"]
Sep 09 20:48:22.596213 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.596173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/858079ab-58e7-459f-be14-de69848ed360-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" (UID: \"858079ab-58e7-459f-be14-de69848ed360\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:22.596213 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.596214    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbwwx\" (UniqueName: \"kubernetes.io/projected/858079ab-58e7-459f-be14-de69848ed360-kube-api-access-hbwwx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" (UID: \"858079ab-58e7-459f-be14-de69848ed360\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:22.697533 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.697477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/858079ab-58e7-459f-be14-de69848ed360-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" (UID: \"858079ab-58e7-459f-be14-de69848ed360\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:22.697533 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.697527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbwwx\" (UniqueName: \"kubernetes.io/projected/858079ab-58e7-459f-be14-de69848ed360-kube-api-access-hbwwx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" (UID: \"858079ab-58e7-459f-be14-de69848ed360\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:22.697851 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.697830    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/858079ab-58e7-459f-be14-de69848ed360-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" (UID: \"858079ab-58e7-459f-be14-de69848ed360\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:22.715538 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.715508    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbwwx\" (UniqueName: \"kubernetes.io/projected/858079ab-58e7-459f-be14-de69848ed360-kube-api-access-hbwwx\") pod \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" (UID: \"858079ab-58e7-459f-be14-de69848ed360\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:22.746246 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:22.746185    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:23.021238 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:23.021158    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:23.477157 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:23.477075    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4gg4f\""
Sep 09 20:48:23.486367 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:23.486352    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:23.772238 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:23.772210    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"]
Sep 09 20:48:23.775738 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:48:23.775702    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod858079ab_58e7_459f_be14_de69848ed360.slice/crio-5ab0eebc934341192c0ba03732bd868eeacb44f65be75fe01834f8b30566649c WatchSource:0}: Error finding container 5ab0eebc934341192c0ba03732bd868eeacb44f65be75fe01834f8b30566649c: Status 404 returned error can't find the container with id 5ab0eebc934341192c0ba03732bd868eeacb44f65be75fe01834f8b30566649c
Sep 09 20:48:23.971879 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:23.971842    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn" event={"ID":"858079ab-58e7-459f-be14-de69848ed360","Type":"ContainerStarted","Data":"5ab0eebc934341192c0ba03732bd868eeacb44f65be75fe01834f8b30566649c"}
Sep 09 20:48:24.776903 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:24.776868    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:24.972116 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:24.972081    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"]
Sep 09 20:48:24.975531 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:24.975509    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"
Sep 09 20:48:24.981011 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:24.980988    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 09 20:48:24.981549 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:24.981528    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-lnz5j\""
Sep 09 20:48:25.008805 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.008779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"]
Sep 09 20:48:25.115003 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.114924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2gj8\" (UniqueName: \"kubernetes.io/projected/4b719d34-29d8-4174-ad70-2232de060462-kube-api-access-k2gj8\") pod \"dns-operator-controller-manager-7b84cd7c9c-skwn8\" (UID: \"4b719d34-29d8-4174-ad70-2232de060462\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"
Sep 09 20:48:25.148176 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.148128    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:25.216107 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.216078    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2gj8\" (UniqueName: \"kubernetes.io/projected/4b719d34-29d8-4174-ad70-2232de060462-kube-api-access-k2gj8\") pod \"dns-operator-controller-manager-7b84cd7c9c-skwn8\" (UID: \"4b719d34-29d8-4174-ad70-2232de060462\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"
Sep 09 20:48:25.239252 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.239217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2gj8\" (UniqueName: \"kubernetes.io/projected/4b719d34-29d8-4174-ad70-2232de060462-kube-api-access-k2gj8\") pod \"dns-operator-controller-manager-7b84cd7c9c-skwn8\" (UID: \"4b719d34-29d8-4174-ad70-2232de060462\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"
Sep 09 20:48:25.288396 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.288364    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"
Sep 09 20:48:25.428370 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.428265    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:25.556560 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.556529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:25.588717 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.588686    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"]
Sep 09 20:48:25.594159 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:48:25.594131    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b719d34_29d8_4174_ad70_2232de060462.slice/crio-715e215799712fae6c44ec577cef7ab5351b049cdc9df688c781f522ad57d6b5 WatchSource:0}: Error finding container 715e215799712fae6c44ec577cef7ab5351b049cdc9df688c781f522ad57d6b5: Status 404 returned error can't find the container with id 715e215799712fae6c44ec577cef7ab5351b049cdc9df688c781f522ad57d6b5
Sep 09 20:48:25.982504 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:25.982469    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8" event={"ID":"4b719d34-29d8-4174-ad70-2232de060462","Type":"ContainerStarted","Data":"715e215799712fae6c44ec577cef7ab5351b049cdc9df688c781f522ad57d6b5"}
Sep 09 20:48:27.310317 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.310283    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-f94rx"]
Sep 09 20:48:27.313725 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.313704    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-f94rx"
Sep 09 20:48:27.316293 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.316275    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-wrzjr\""
Sep 09 20:48:27.329986 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.329965    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-f94rx"]
Sep 09 20:48:27.438989 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.438951    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgwx5\" (UniqueName: \"kubernetes.io/projected/e4813fee-7416-4e1c-a7ff-8f990a2613b1-kube-api-access-dgwx5\") pod \"authorino-operator-76b478bd78-f94rx\" (UID: \"e4813fee-7416-4e1c-a7ff-8f990a2613b1\") " pod="kuadrant-system/authorino-operator-76b478bd78-f94rx"
Sep 09 20:48:27.539805 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.539766    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgwx5\" (UniqueName: \"kubernetes.io/projected/e4813fee-7416-4e1c-a7ff-8f990a2613b1-kube-api-access-dgwx5\") pod \"authorino-operator-76b478bd78-f94rx\" (UID: \"e4813fee-7416-4e1c-a7ff-8f990a2613b1\") " pod="kuadrant-system/authorino-operator-76b478bd78-f94rx"
Sep 09 20:48:27.552748 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.552724    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgwx5\" (UniqueName: \"kubernetes.io/projected/e4813fee-7416-4e1c-a7ff-8f990a2613b1-kube-api-access-dgwx5\") pod \"authorino-operator-76b478bd78-f94rx\" (UID: \"e4813fee-7416-4e1c-a7ff-8f990a2613b1\") " pod="kuadrant-system/authorino-operator-76b478bd78-f94rx"
Sep 09 20:48:27.625417 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.625392    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-f94rx"
Sep 09 20:48:27.917440 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.917359    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-f94rx"]
Sep 09 20:48:27.921200 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:48:27.921168    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4813fee_7416_4e1c_a7ff_8f990a2613b1.slice/crio-d4ff6c0932a25a1586e6b0526c221c7f16ef22d3e9d942550d42f338bfc8496f WatchSource:0}: Error finding container d4ff6c0932a25a1586e6b0526c221c7f16ef22d3e9d942550d42f338bfc8496f: Status 404 returned error can't find the container with id d4ff6c0932a25a1586e6b0526c221c7f16ef22d3e9d942550d42f338bfc8496f
Sep 09 20:48:27.993405 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.993369    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-f94rx" event={"ID":"e4813fee-7416-4e1c-a7ff-8f990a2613b1","Type":"ContainerStarted","Data":"d4ff6c0932a25a1586e6b0526c221c7f16ef22d3e9d942550d42f338bfc8496f"}
Sep 09 20:48:27.995254 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.995228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn" event={"ID":"858079ab-58e7-459f-be14-de69848ed360","Type":"ContainerStarted","Data":"bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222"}
Sep 09 20:48:27.995469 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:27.995447    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:28.049874 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:28.046492    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn" podStartSLOduration=2.270597039 podStartE2EDuration="6.045072722s" totalImagesPullingTime="3.774475683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:28.041082195 +0000 UTC m=+488.232919794" watchObservedRunningTime="2026-09-09 20:48:28.045072722 +0000 UTC m=+488.236910322"
Sep 09 20:48:28.564995 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:28.564956    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:28.672837 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:28.672803    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:29.001688 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:29.001641    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8" event={"ID":"4b719d34-29d8-4174-ad70-2232de060462","Type":"ContainerStarted","Data":"2a554a0c92022a14eadb23748f7451678f325d8e1a0090f41f9eb8b00b566ed1"}
Sep 09 20:48:29.001929 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:29.001905    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"
Sep 09 20:48:29.048570 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:29.048516    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8" podStartSLOduration=2.209609971 podStartE2EDuration="5.048500083s" totalImagesPullingTime="2.838890112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:29.047027024 +0000 UTC m=+489.238864635" watchObservedRunningTime="2026-09-09 20:48:29.048500083 +0000 UTC m=+489.240337681"
Sep 09 20:48:30.007322 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:30.007270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-f94rx" event={"ID":"e4813fee-7416-4e1c-a7ff-8f990a2613b1","Type":"ContainerStarted","Data":"421fcc5c225bd73facfbdb3bb9ec608b3ffc4c047ae46085fbf454f8f02f55f6"}
Sep 09 20:48:30.039435 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:30.039389    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-f94rx" podStartSLOduration=1.4501727309999999 podStartE2EDuration="3.039374216s" totalImagesPullingTime="1.589201485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:30.037582791 +0000 UTC m=+490.229420389" watchObservedRunningTime="2026-09-09 20:48:30.039374216 +0000 UTC m=+490.231211815"
Sep 09 20:48:31.011950 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.011921    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-f94rx"
Sep 09 20:48:31.181360 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.181329    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:31.261600 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.261568    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"]
Sep 09 20:48:31.264688 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.264646    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.271797 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.271781    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 09 20:48:31.274081 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.274068    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wzwpv\""
Sep 09 20:48:31.274272 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.274256    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 09 20:48:31.291202 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.291180    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"]
Sep 09 20:48:31.367827 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.367798    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2476a5e6-8c10-4cd5-a581-c52840309c25-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-q58hc\" (UID: \"2476a5e6-8c10-4cd5-a581-c52840309c25\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.367961 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.367839    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2476a5e6-8c10-4cd5-a581-c52840309c25-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-q58hc\" (UID: \"2476a5e6-8c10-4cd5-a581-c52840309c25\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.367961 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.367864    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ms5f4\" (UniqueName: \"kubernetes.io/projected/2476a5e6-8c10-4cd5-a581-c52840309c25-kube-api-access-ms5f4\") pod \"kuadrant-console-plugin-fbdd4f445-q58hc\" (UID: \"2476a5e6-8c10-4cd5-a581-c52840309c25\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.461606 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.461577    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:31.468572 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.468541    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2476a5e6-8c10-4cd5-a581-c52840309c25-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-q58hc\" (UID: \"2476a5e6-8c10-4cd5-a581-c52840309c25\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.468673 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.468605    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2476a5e6-8c10-4cd5-a581-c52840309c25-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-q58hc\" (UID: \"2476a5e6-8c10-4cd5-a581-c52840309c25\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.468673 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.468637    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ms5f4\" (UniqueName: \"kubernetes.io/projected/2476a5e6-8c10-4cd5-a581-c52840309c25-kube-api-access-ms5f4\") pod \"kuadrant-console-plugin-fbdd4f445-q58hc\" (UID: \"2476a5e6-8c10-4cd5-a581-c52840309c25\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.469331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.469288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2476a5e6-8c10-4cd5-a581-c52840309c25-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-q58hc\" (UID: \"2476a5e6-8c10-4cd5-a581-c52840309c25\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.470870 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.470853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/2476a5e6-8c10-4cd5-a581-c52840309c25-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-q58hc\" (UID: \"2476a5e6-8c10-4cd5-a581-c52840309c25\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.478489 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.478467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ms5f4\" (UniqueName: \"kubernetes.io/projected/2476a5e6-8c10-4cd5-a581-c52840309c25-kube-api-access-ms5f4\") pod \"kuadrant-console-plugin-fbdd4f445-q58hc\" (UID: \"2476a5e6-8c10-4cd5-a581-c52840309c25\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.573041 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.572953    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"
Sep 09 20:48:31.852094 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.852017    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc"]
Sep 09 20:48:31.855521 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:48:31.855493    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2476a5e6_8c10_4cd5_a581_c52840309c25.slice/crio-47e4631e7bf4f2f2e8f5b234e2fad01ee305d48327c6ae032c678ce306ba0d21 WatchSource:0}: Error finding container 47e4631e7bf4f2f2e8f5b234e2fad01ee305d48327c6ae032c678ce306ba0d21: Status 404 returned error can't find the container with id 47e4631e7bf4f2f2e8f5b234e2fad01ee305d48327c6ae032c678ce306ba0d21
Sep 09 20:48:31.965266 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:31.965243    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:32.022915 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:32.022879    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc" event={"ID":"2476a5e6-8c10-4cd5-a581-c52840309c25","Type":"ContainerStarted","Data":"47e4631e7bf4f2f2e8f5b234e2fad01ee305d48327c6ae032c678ce306ba0d21"}
Sep 09 20:48:32.546886 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:32.546855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:32.742901 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:32.742870    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:32.928986 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:32.928900    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:33.191014 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:33.190927    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:37.050678 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:37.050648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc" event={"ID":"2476a5e6-8c10-4cd5-a581-c52840309c25","Type":"ContainerStarted","Data":"86c59fd068635c375b0f6960a0763e408d499cc36bd2eb0d4c28100db0be866e"}
Sep 09 20:48:37.073290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:37.073244    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-q58hc" podStartSLOduration=1.337195167 podStartE2EDuration="6.073228185s" totalImagesPullingTime="4.736033018s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:37.073096845 +0000 UTC m=+497.264934427" watchObservedRunningTime="2026-09-09 20:48:37.073228185 +0000 UTC m=+497.265065785"
Sep 09 20:48:39.004683 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:39.004651    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:39.012906 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:39.012878    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:39.115112 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:39.115085    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:40.009302 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:40.009274    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-skwn8"
Sep 09 20:48:41.421775 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.421741    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"]
Sep 09 20:48:41.422156 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.421953    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn" podUID="858079ab-58e7-459f-be14-de69848ed360" containerName="manager" containerID="cri-o://bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222" gracePeriod=2
Sep 09 20:48:41.429280 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.429255    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"]
Sep 09 20:48:41.459998 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.459969    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"]
Sep 09 20:48:41.460251 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.460200    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" containerName="manager" containerID="cri-o://cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf" gracePeriod=2
Sep 09 20:48:41.462200 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.462175    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"]
Sep 09 20:48:41.462625 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.462604    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="858079ab-58e7-459f-be14-de69848ed360" containerName="manager"
Sep 09 20:48:41.462683 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.462631    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="858079ab-58e7-459f-be14-de69848ed360" containerName="manager"
Sep 09 20:48:41.462726 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.462715    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="858079ab-58e7-459f-be14-de69848ed360" containerName="manager"
Sep 09 20:48:41.464574 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:41.464541    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:41.465982 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.465962    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:41.473364 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.473341    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"]
Sep 09 20:48:41.484132 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:41.484108    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7f85j\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:41.494694 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.494671    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"]
Sep 09 20:48:41.508795 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.508776    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"]
Sep 09 20:48:41.509051 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.509039    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" containerName="manager"
Sep 09 20:48:41.509104 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.509054    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" containerName="manager"
Sep 09 20:48:41.509140 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.509122    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" containerName="manager"
Sep 09 20:48:41.511848 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.511835    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"
Sep 09 20:48:41.517776 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:41.517740    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:41.519619 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:41.519600    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7f85j\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:41.532384 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.532364    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"]
Sep 09 20:48:41.581533 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:41.581492    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:41.648878 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.648845    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ns62\" (UniqueName: \"kubernetes.io/projected/b4da00f0-8115-4524-ac31-eca7c1fa376a-kube-api-access-5ns62\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tq9tw\" (UID: \"b4da00f0-8115-4524-ac31-eca7c1fa376a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:41.648984 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.648882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8x76\" (UniqueName: \"kubernetes.io/projected/a93f16ac-0405-42b4-8407-b551870fefd5-kube-api-access-m8x76\") pod \"limitador-operator-controller-manager-74495c69f8-dfktn\" (UID: \"a93f16ac-0405-42b4-8407-b551870fefd5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"
Sep 09 20:48:41.649030 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.649002    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4da00f0-8115-4524-ac31-eca7c1fa376a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tq9tw\" (UID: \"b4da00f0-8115-4524-ac31-eca7c1fa376a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:41.691102 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.691082    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:41.693364 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:41.693333    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:41.693773 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.693755    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:41.695802 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:41.695780    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7f85j\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:41.697756 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:41.697737    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:41.700959 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:41.700939    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7f85j\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:41.749499 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.749476    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ns62\" (UniqueName: \"kubernetes.io/projected/b4da00f0-8115-4524-ac31-eca7c1fa376a-kube-api-access-5ns62\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tq9tw\" (UID: \"b4da00f0-8115-4524-ac31-eca7c1fa376a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:41.749499 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.749505    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8x76\" (UniqueName: \"kubernetes.io/projected/a93f16ac-0405-42b4-8407-b551870fefd5-kube-api-access-m8x76\") pod \"limitador-operator-controller-manager-74495c69f8-dfktn\" (UID: \"a93f16ac-0405-42b4-8407-b551870fefd5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"
Sep 09 20:48:41.749628 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.749550    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4da00f0-8115-4524-ac31-eca7c1fa376a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tq9tw\" (UID: \"b4da00f0-8115-4524-ac31-eca7c1fa376a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:41.749828 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.749814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4da00f0-8115-4524-ac31-eca7c1fa376a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tq9tw\" (UID: \"b4da00f0-8115-4524-ac31-eca7c1fa376a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:41.768296 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.768275    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8x76\" (UniqueName: \"kubernetes.io/projected/a93f16ac-0405-42b4-8407-b551870fefd5-kube-api-access-m8x76\") pod \"limitador-operator-controller-manager-74495c69f8-dfktn\" (UID: \"a93f16ac-0405-42b4-8407-b551870fefd5\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"
Sep 09 20:48:41.768539 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.768519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ns62\" (UniqueName: \"kubernetes.io/projected/b4da00f0-8115-4524-ac31-eca7c1fa376a-kube-api-access-5ns62\") pod \"kuadrant-operator-controller-manager-76f8f686bf-tq9tw\" (UID: \"b4da00f0-8115-4524-ac31-eca7c1fa376a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:41.850466 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.850444    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/858079ab-58e7-459f-be14-de69848ed360-extensions-socket-volume\") pod \"858079ab-58e7-459f-be14-de69848ed360\" (UID: \"858079ab-58e7-459f-be14-de69848ed360\") "
Sep 09 20:48:41.850550 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.850486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/5170e3d5-cb80-4c5d-b2d8-cad4f616c46f-kube-api-access-mb66c\") pod \"5170e3d5-cb80-4c5d-b2d8-cad4f616c46f\" (UID: \"5170e3d5-cb80-4c5d-b2d8-cad4f616c46f\") "
Sep 09 20:48:41.850550 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.850520    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbwwx\" (UniqueName: \"kubernetes.io/projected/858079ab-58e7-459f-be14-de69848ed360-kube-api-access-hbwwx\") pod \"858079ab-58e7-459f-be14-de69848ed360\" (UID: \"858079ab-58e7-459f-be14-de69848ed360\") "
Sep 09 20:48:41.850898 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.850877    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/858079ab-58e7-459f-be14-de69848ed360-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "858079ab-58e7-459f-be14-de69848ed360" (UID: "858079ab-58e7-459f-be14-de69848ed360"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:48:41.852460 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.852441    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5170e3d5-cb80-4c5d-b2d8-cad4f616c46f-kube-api-access-mb66c" (OuterVolumeSpecName: "kube-api-access-mb66c") pod "5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" (UID: "5170e3d5-cb80-4c5d-b2d8-cad4f616c46f"). InnerVolumeSpecName "kube-api-access-mb66c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:48:41.852538 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.852499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858079ab-58e7-459f-be14-de69848ed360-kube-api-access-hbwwx" (OuterVolumeSpecName: "kube-api-access-hbwwx") pod "858079ab-58e7-459f-be14-de69848ed360" (UID: "858079ab-58e7-459f-be14-de69848ed360"). InnerVolumeSpecName "kube-api-access-hbwwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:48:41.854647 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.854633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:41.861886 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.861863    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"
Sep 09 20:48:41.951858 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.951831    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/858079ab-58e7-459f-be14-de69848ed360-extensions-socket-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:48:41.951858 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.951859    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb66c\" (UniqueName: \"kubernetes.io/projected/5170e3d5-cb80-4c5d-b2d8-cad4f616c46f-kube-api-access-mb66c\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:48:41.951993 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:41.951870    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbwwx\" (UniqueName: \"kubernetes.io/projected/858079ab-58e7-459f-be14-de69848ed360-kube-api-access-hbwwx\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:48:42.025649 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.025623    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-f94rx"
Sep 09 20:48:42.067600 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.067570    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:42.070395 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.070360    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7f85j\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:42.072396 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.072375    2566 generic.go:356] "Generic (PLEG): container finished" podID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" containerID="cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf" exitCode=0
Sep 09 20:48:42.072482 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.072416    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:42.072482 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.072470    2566 scope.go:117] "RemoveContainer" containerID="cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf"
Sep 09 20:48:42.074058 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.074032    2566 generic.go:356] "Generic (PLEG): container finished" podID="858079ab-58e7-459f-be14-de69848ed360" containerID="bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222" exitCode=0
Sep 09 20:48:42.074131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.074071    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:42.079711 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.075465    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:42.079889 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.079861    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7f85j\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:42.082848 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.082639    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:42.084544 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.084516    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7f85j\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:42.085524 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.085503    2566 scope.go:117] "RemoveContainer" containerID="cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf"
Sep 09 20:48:42.085835 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.085814    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf\": container with ID starting with cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf not found: ID does not exist" containerID="cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf"
Sep 09 20:48:42.085928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.085843    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf"} err="failed to get container status \"cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf\": rpc error: code = NotFound desc = could not find container \"cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf\": container with ID starting with cdb5a3ca5c87a0ef595a88d6274080e437debb013799a7913937b443b8009ddf not found: ID does not exist"
Sep 09 20:48:42.085928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.085866    2566 scope.go:117] "RemoveContainer" containerID="bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222"
Sep 09 20:48:42.090419 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.090393    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7f85j\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:42.092626 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.092597    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:42.093765 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.093745    2566 scope.go:117] "RemoveContainer" containerID="bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222"
Sep 09 20:48:42.094055 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.094035    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222\": container with ID starting with bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222 not found: ID does not exist" containerID="bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222"
Sep 09 20:48:42.094122 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.094061    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222"} err="failed to get container status \"bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222\": rpc error: code = NotFound desc = could not find container \"bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222\": container with ID starting with bdbef51d56a605139aa9ba5dafe36e3ef194f6f5c63ebe0185143640c10b6222 not found: ID does not exist"
Sep 09 20:48:42.094424 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.094402    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-4zxqn\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="858079ab-58e7-459f-be14-de69848ed360" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-4zxqn"
Sep 09 20:48:42.096198 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:48:42.096178    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-7f85j\" is forbidden: User \"system:node:ip-10-0-137-187.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-187.ec2.internal' and this object" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-7f85j"
Sep 09 20:48:42.150135 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.150105    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"]
Sep 09 20:48:42.151644 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.151623    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"]
Sep 09 20:48:42.153275 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:48:42.153250    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4da00f0_8115_4524_ac31_eca7c1fa376a.slice/crio-ba730c676f490bfc66e41875e0bf866812680528cf96e856a3dc6acf227fe185 WatchSource:0}: Error finding container ba730c676f490bfc66e41875e0bf866812680528cf96e856a3dc6acf227fe185: Status 404 returned error can't find the container with id ba730c676f490bfc66e41875e0bf866812680528cf96e856a3dc6acf227fe185
Sep 09 20:48:42.153911 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:48:42.153890    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93f16ac_0405_42b4_8407_b551870fefd5.slice/crio-5beae46027d0e6ca942fdb74055b5223d77269d1ab6bf5e023a3c5ade56c75c4 WatchSource:0}: Error finding container 5beae46027d0e6ca942fdb74055b5223d77269d1ab6bf5e023a3c5ade56c75c4: Status 404 returned error can't find the container with id 5beae46027d0e6ca942fdb74055b5223d77269d1ab6bf5e023a3c5ade56c75c4
Sep 09 20:48:42.285142 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.285106    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5170e3d5-cb80-4c5d-b2d8-cad4f616c46f" path="/var/lib/kubelet/pods/5170e3d5-cb80-4c5d-b2d8-cad4f616c46f/volumes"
Sep 09 20:48:42.285537 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:42.285518    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="858079ab-58e7-459f-be14-de69848ed360" path="/var/lib/kubelet/pods/858079ab-58e7-459f-be14-de69848ed360/volumes"
Sep 09 20:48:43.080624 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:43.080583    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn" event={"ID":"a93f16ac-0405-42b4-8407-b551870fefd5","Type":"ContainerStarted","Data":"6776182b1a98523373a9610306811ecd52acc283c065bac50a6fdbb3722f1d87"}
Sep 09 20:48:43.081040 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:43.080644    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn" event={"ID":"a93f16ac-0405-42b4-8407-b551870fefd5","Type":"ContainerStarted","Data":"5beae46027d0e6ca942fdb74055b5223d77269d1ab6bf5e023a3c5ade56c75c4"}
Sep 09 20:48:43.081040 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:43.080672    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"
Sep 09 20:48:43.083292 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:43.083269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw" event={"ID":"b4da00f0-8115-4524-ac31-eca7c1fa376a","Type":"ContainerStarted","Data":"155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e"}
Sep 09 20:48:43.083380 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:43.083337    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:43.083380 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:43.083356    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw" event={"ID":"b4da00f0-8115-4524-ac31-eca7c1fa376a","Type":"ContainerStarted","Data":"ba730c676f490bfc66e41875e0bf866812680528cf96e856a3dc6acf227fe185"}
Sep 09 20:48:43.139844 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:43.139801    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn" podStartSLOduration=2.139787177 podStartE2EDuration="2.139787177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:43.13694102 +0000 UTC m=+503.328778620" watchObservedRunningTime="2026-09-09 20:48:43.139787177 +0000 UTC m=+503.331624775"
Sep 09 20:48:43.234819 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:43.234776    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw" podStartSLOduration=2.234762782 podStartE2EDuration="2.234762782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:48:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:48:43.233936418 +0000 UTC m=+503.425774050" watchObservedRunningTime="2026-09-09 20:48:43.234762782 +0000 UTC m=+503.426600380"
Sep 09 20:48:45.552941 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:45.552907    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:45.687938 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:45.687905    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:47.525871 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:47.525834    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:47.616853 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:47.616820    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:48.131898 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:48.131860    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:48.246662 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:48.246624    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:51.925958 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:51.925905    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:52.034666 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:52.034631    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:52.115033 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:52.114986    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:52.239509 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:52.239474    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:52.313344 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:52.313297    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:52.413187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:52.413161    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:54.090404 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:54.090376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:48:54.090810 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:54.090438    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-dfktn"
Sep 09 20:48:58.407531 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:58.407493    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:48:58.539761 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:48:58.539727    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:00.042349 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:00.042294    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"]
Sep 09 20:49:00.042805 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:00.042554    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw" podUID="b4da00f0-8115-4524-ac31-eca7c1fa376a" containerName="manager" containerID="cri-o://155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e" gracePeriod=10
Sep 09 20:49:00.794952 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:00.794928    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:49:00.893062 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:00.893019    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5ns62\" (UniqueName: \"kubernetes.io/projected/b4da00f0-8115-4524-ac31-eca7c1fa376a-kube-api-access-5ns62\") pod \"b4da00f0-8115-4524-ac31-eca7c1fa376a\" (UID: \"b4da00f0-8115-4524-ac31-eca7c1fa376a\") "
Sep 09 20:49:00.893228 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:00.893079    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4da00f0-8115-4524-ac31-eca7c1fa376a-extensions-socket-volume\") pod \"b4da00f0-8115-4524-ac31-eca7c1fa376a\" (UID: \"b4da00f0-8115-4524-ac31-eca7c1fa376a\") "
Sep 09 20:49:00.893510 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:00.893460    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4da00f0-8115-4524-ac31-eca7c1fa376a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b4da00f0-8115-4524-ac31-eca7c1fa376a" (UID: "b4da00f0-8115-4524-ac31-eca7c1fa376a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:49:00.895146 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:00.895120    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4da00f0-8115-4524-ac31-eca7c1fa376a-kube-api-access-5ns62" (OuterVolumeSpecName: "kube-api-access-5ns62") pod "b4da00f0-8115-4524-ac31-eca7c1fa376a" (UID: "b4da00f0-8115-4524-ac31-eca7c1fa376a"). InnerVolumeSpecName "kube-api-access-5ns62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:00.994340 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:00.994290    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b4da00f0-8115-4524-ac31-eca7c1fa376a-extensions-socket-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:49:00.994340 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:00.994335    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ns62\" (UniqueName: \"kubernetes.io/projected/b4da00f0-8115-4524-ac31-eca7c1fa376a-kube-api-access-5ns62\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:49:01.161523 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:01.161486    2566 generic.go:356] "Generic (PLEG): container finished" podID="b4da00f0-8115-4524-ac31-eca7c1fa376a" containerID="155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e" exitCode=0
Sep 09 20:49:01.161940 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:01.161537    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"
Sep 09 20:49:01.161940 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:01.161574    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw" event={"ID":"b4da00f0-8115-4524-ac31-eca7c1fa376a","Type":"ContainerDied","Data":"155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e"}
Sep 09 20:49:01.161940 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:01.161634    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw" event={"ID":"b4da00f0-8115-4524-ac31-eca7c1fa376a","Type":"ContainerDied","Data":"ba730c676f490bfc66e41875e0bf866812680528cf96e856a3dc6acf227fe185"}
Sep 09 20:49:01.161940 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:01.161663    2566 scope.go:117] "RemoveContainer" containerID="155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e"
Sep 09 20:49:01.170245 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:01.170228    2566 scope.go:117] "RemoveContainer" containerID="155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e"
Sep 09 20:49:01.170529 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:49:01.170511    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e\": container with ID starting with 155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e not found: ID does not exist" containerID="155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e"
Sep 09 20:49:01.170591 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:01.170537    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e"} err="failed to get container status \"155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e\": rpc error: code = NotFound desc = could not find container \"155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e\": container with ID starting with 155da5486a1b3e9a1d98cdcb313350e7e6a44f5a811023d4b5bf17d2ddddd79e not found: ID does not exist"
Sep 09 20:49:01.187541 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:01.187517    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"]
Sep 09 20:49:01.191835 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:01.191814    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-tq9tw"]
Sep 09 20:49:02.285290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:02.285249    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4da00f0-8115-4524-ac31-eca7c1fa376a" path="/var/lib/kubelet/pods/b4da00f0-8115-4524-ac31-eca7c1fa376a/volumes"
Sep 09 20:49:04.932938 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:04.932903    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:05.034382 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:05.034348    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:13.128082 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:13.128044    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:16.455695 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.455659    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"]
Sep 09 20:49:16.456146 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.456062    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4da00f0-8115-4524-ac31-eca7c1fa376a" containerName="manager"
Sep 09 20:49:16.456146 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.456079    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4da00f0-8115-4524-ac31-eca7c1fa376a" containerName="manager"
Sep 09 20:49:16.456257 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.456155    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4da00f0-8115-4524-ac31-eca7c1fa376a" containerName="manager"
Sep 09 20:49:16.457990 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.457970    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.460584 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.460562    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 09 20:49:16.460711 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.460600    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-5z6z6\""
Sep 09 20:49:16.472463 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.472440    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"]
Sep 09 20:49:16.502820 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.502793    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.502953 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.502828    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/712d1861-3ac0-4548-9336-953dc2d2af00-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.502953 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.502848    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.502953 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.502878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/712d1861-3ac0-4548-9336-953dc2d2af00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.502953 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.502938    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.502953 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.502953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.503198 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.502976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.503198 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.503041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sn44\" (UniqueName: \"kubernetes.io/projected/712d1861-3ac0-4548-9336-953dc2d2af00-kube-api-access-9sn44\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.503198 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.503100    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/712d1861-3ac0-4548-9336-953dc2d2af00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.516236 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.516209    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:16.604363 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604327    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604363 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604365    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604595 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604395    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604595 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604416    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sn44\" (UniqueName: \"kubernetes.io/projected/712d1861-3ac0-4548-9336-953dc2d2af00-kube-api-access-9sn44\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604595 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604437    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/712d1861-3ac0-4548-9336-953dc2d2af00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604595 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604468    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604595 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604502    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/712d1861-3ac0-4548-9336-953dc2d2af00-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604595 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604527    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604595 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/712d1861-3ac0-4548-9336-953dc2d2af00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604943 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.604943 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604854    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.605045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.604948    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.605114 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.605052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.605257 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.605231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/712d1861-3ac0-4548-9336-953dc2d2af00-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.606868 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.606844    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/712d1861-3ac0-4548-9336-953dc2d2af00-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.606945 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.606909    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/712d1861-3ac0-4548-9336-953dc2d2af00-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.613293 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.613257    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/712d1861-3ac0-4548-9336-953dc2d2af00-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.613479 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.613459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sn44\" (UniqueName: \"kubernetes.io/projected/712d1861-3ac0-4548-9336-953dc2d2af00-kube-api-access-9sn44\") pod \"maas-default-gateway-openshift-default-78bb78ff78-7mxfq\" (UID: \"712d1861-3ac0-4548-9336-953dc2d2af00\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.768185 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.768157    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:16.866588 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:16.866551    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:17.049382 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:17.049288    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"]
Sep 09 20:49:17.052123 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:49:17.052096    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712d1861_3ac0_4548_9336_953dc2d2af00.slice/crio-e66f2129df592468153148671946735db18fd5de379aee0064fa91905529e80f WatchSource:0}: Error finding container e66f2129df592468153148671946735db18fd5de379aee0064fa91905529e80f: Status 404 returned error can't find the container with id e66f2129df592468153148671946735db18fd5de379aee0064fa91905529e80f
Sep 09 20:49:17.231140 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:17.231102    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" event={"ID":"712d1861-3ac0-4548-9336-953dc2d2af00","Type":"ContainerStarted","Data":"e66f2129df592468153148671946735db18fd5de379aee0064fa91905529e80f"}
Sep 09 20:49:19.641630 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:19.641597    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:49:19.641864 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:19.641667    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:49:19.641864 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:19.641698    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:49:20.247025 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:20.246992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" event={"ID":"712d1861-3ac0-4548-9336-953dc2d2af00","Type":"ContainerStarted","Data":"e79ffdd33f2e51b242978a40311843f08661c84a57c151f9cb28b171d2d1ead2"}
Sep 09 20:49:20.272478 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:20.272439    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" podStartSLOduration=1.685247694 podStartE2EDuration="4.272425335s" totalImagesPullingTime="2.587177641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:20.270803384 +0000 UTC m=+540.462640983" watchObservedRunningTime="2026-09-09 20:49:20.272425335 +0000 UTC m=+540.464262933"
Sep 09 20:49:20.769361 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:20.769333    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:20.773882 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:20.773859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:21.258288 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.258255    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:21.259355 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.259333    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:49:21.351440 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.351412    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mm4dg"]
Sep 09 20:49:21.353939 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.353923    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:21.355897 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.355868    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:21.356225 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.356209    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 09 20:49:21.365755 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.365730    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mm4dg"]
Sep 09 20:49:21.441871 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.441842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33f95c14-6d03-478c-bc70-f62c73eb9b61-config-file\") pod \"limitador-limitador-8588749977-mm4dg\" (UID: \"33f95c14-6d03-478c-bc70-f62c73eb9b61\") " pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:21.442027 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.441899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rfzc4\" (UniqueName: \"kubernetes.io/projected/33f95c14-6d03-478c-bc70-f62c73eb9b61-kube-api-access-rfzc4\") pod \"limitador-limitador-8588749977-mm4dg\" (UID: \"33f95c14-6d03-478c-bc70-f62c73eb9b61\") " pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:21.446921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.446896    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mm4dg"]
Sep 09 20:49:21.450081 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.450040    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:21.479581 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.479554    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:49:21.481584 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.481569    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:21.496245 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.496223    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:49:21.525172 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.525092    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:49:21.526733 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.526708    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:21.542718 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.542694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmpvr\" (UniqueName: \"kubernetes.io/projected/d56b66e0-bc47-47f3-92bc-78ce181afb88-kube-api-access-tmpvr\") pod \"limitador-limitador-84994f7ffc-gkth4\" (UID: \"d56b66e0-bc47-47f3-92bc-78ce181afb88\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:21.542843 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.542741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rfzc4\" (UniqueName: \"kubernetes.io/projected/33f95c14-6d03-478c-bc70-f62c73eb9b61-kube-api-access-rfzc4\") pod \"limitador-limitador-8588749977-mm4dg\" (UID: \"33f95c14-6d03-478c-bc70-f62c73eb9b61\") " pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:21.542843 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.542795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d56b66e0-bc47-47f3-92bc-78ce181afb88-config-file\") pod \"limitador-limitador-84994f7ffc-gkth4\" (UID: \"d56b66e0-bc47-47f3-92bc-78ce181afb88\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:21.542933 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.542865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33f95c14-6d03-478c-bc70-f62c73eb9b61-config-file\") pod \"limitador-limitador-8588749977-mm4dg\" (UID: \"33f95c14-6d03-478c-bc70-f62c73eb9b61\") " pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:21.543496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.543475    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33f95c14-6d03-478c-bc70-f62c73eb9b61-config-file\") pod \"limitador-limitador-8588749977-mm4dg\" (UID: \"33f95c14-6d03-478c-bc70-f62c73eb9b61\") " pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:21.551117 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.551095    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rfzc4\" (UniqueName: \"kubernetes.io/projected/33f95c14-6d03-478c-bc70-f62c73eb9b61-kube-api-access-rfzc4\") pod \"limitador-limitador-8588749977-mm4dg\" (UID: \"33f95c14-6d03-478c-bc70-f62c73eb9b61\") " pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:21.643233 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.643203    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmpvr\" (UniqueName: \"kubernetes.io/projected/d56b66e0-bc47-47f3-92bc-78ce181afb88-kube-api-access-tmpvr\") pod \"limitador-limitador-84994f7ffc-gkth4\" (UID: \"d56b66e0-bc47-47f3-92bc-78ce181afb88\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:21.643378 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.643249    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d56b66e0-bc47-47f3-92bc-78ce181afb88-config-file\") pod \"limitador-limitador-84994f7ffc-gkth4\" (UID: \"d56b66e0-bc47-47f3-92bc-78ce181afb88\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:21.643921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.643903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d56b66e0-bc47-47f3-92bc-78ce181afb88-config-file\") pod \"limitador-limitador-84994f7ffc-gkth4\" (UID: \"d56b66e0-bc47-47f3-92bc-78ce181afb88\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:21.651763 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.651741    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmpvr\" (UniqueName: \"kubernetes.io/projected/d56b66e0-bc47-47f3-92bc-78ce181afb88-kube-api-access-tmpvr\") pod \"limitador-limitador-84994f7ffc-gkth4\" (UID: \"d56b66e0-bc47-47f3-92bc-78ce181afb88\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:21.663496 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.663478    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:21.791147 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.791068    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:21.933681 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:21.933657    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mm4dg"]
Sep 09 20:49:21.935767 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:49:21.935738    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33f95c14_6d03_478c_bc70_f62c73eb9b61.slice/crio-5e839b6569f203060c66583f6abcd536e866141d366e6df14514d3bf91f45fef WatchSource:0}: Error finding container 5e839b6569f203060c66583f6abcd536e866141d366e6df14514d3bf91f45fef: Status 404 returned error can't find the container with id 5e839b6569f203060c66583f6abcd536e866141d366e6df14514d3bf91f45fef
Sep 09 20:49:22.064123 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:22.064097    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:49:22.066477 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:49:22.066447    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56b66e0_bc47_47f3_92bc_78ce181afb88.slice/crio-ae2324990158542c9be6535afa84e613932796597a82639e6b01e3437315ae7e WatchSource:0}: Error finding container ae2324990158542c9be6535afa84e613932796597a82639e6b01e3437315ae7e: Status 404 returned error can't find the container with id ae2324990158542c9be6535afa84e613932796597a82639e6b01e3437315ae7e
Sep 09 20:49:22.181575 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:22.181544    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:22.263117 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:22.263084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4" event={"ID":"d56b66e0-bc47-47f3-92bc-78ce181afb88","Type":"ContainerStarted","Data":"ae2324990158542c9be6535afa84e613932796597a82639e6b01e3437315ae7e"}
Sep 09 20:49:22.264299 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:22.264270    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg" event={"ID":"33f95c14-6d03-478c-bc70-f62c73eb9b61","Type":"ContainerStarted","Data":"5e839b6569f203060c66583f6abcd536e866141d366e6df14514d3bf91f45fef"}
Sep 09 20:49:22.277600 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:22.277572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:23.270498 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:23.270465    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4" event={"ID":"d56b66e0-bc47-47f3-92bc-78ce181afb88","Type":"ContainerStarted","Data":"5d08e3f9c8db59be5b08665dd5c6ed78619dd208284a117076dc7f8b37321b8d"}
Sep 09 20:49:23.270907 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:23.270548    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:23.272083 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:23.272058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg" event={"ID":"33f95c14-6d03-478c-bc70-f62c73eb9b61","Type":"ContainerStarted","Data":"a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8"}
Sep 09 20:49:23.287907 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:23.287865    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4" podStartSLOduration=1.456529156 podStartE2EDuration="2.287850912s" totalImagesPullingTime="831.321756ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:23.286596666 +0000 UTC m=+543.478434264" watchObservedRunningTime="2026-09-09 20:49:23.287850912 +0000 UTC m=+543.479688511"
Sep 09 20:49:23.302860 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:23.302817    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg" podStartSLOduration=1.3419273760000001 podStartE2EDuration="2.302804529s" totalImagesPullingTime="960.877153ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:49:23.30152956 +0000 UTC m=+543.493367158" watchObservedRunningTime="2026-09-09 20:49:23.302804529 +0000 UTC m=+543.494642128"
Sep 09 20:49:24.276033 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:24.276004    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:34.277617 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:34.277570    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-gkth4"
Sep 09 20:49:34.325935 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:34.325903    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mm4dg"]
Sep 09 20:49:34.326091 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:34.326062    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg" podUID="33f95c14-6d03-478c-bc70-f62c73eb9b61" containerName="limitador" containerID="cri-o://a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8" gracePeriod=30
Sep 09 20:49:34.326788 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:34.326768    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:35.258041 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.258014    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:35.321765 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.321735    2566 generic.go:356] "Generic (PLEG): container finished" podID="33f95c14-6d03-478c-bc70-f62c73eb9b61" containerID="a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8" exitCode=0
Sep 09 20:49:35.322163 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.321779    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg"
Sep 09 20:49:35.322163 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.321828    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg" event={"ID":"33f95c14-6d03-478c-bc70-f62c73eb9b61","Type":"ContainerDied","Data":"a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8"}
Sep 09 20:49:35.322163 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.321882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-mm4dg" event={"ID":"33f95c14-6d03-478c-bc70-f62c73eb9b61","Type":"ContainerDied","Data":"5e839b6569f203060c66583f6abcd536e866141d366e6df14514d3bf91f45fef"}
Sep 09 20:49:35.322163 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.321917    2566 scope.go:117] "RemoveContainer" containerID="a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8"
Sep 09 20:49:35.329709 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.329549    2566 scope.go:117] "RemoveContainer" containerID="a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8"
Sep 09 20:49:35.329847 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:49:35.329828    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8\": container with ID starting with a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8 not found: ID does not exist" containerID="a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8"
Sep 09 20:49:35.329883 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.329856    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8"} err="failed to get container status \"a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8\": rpc error: code = NotFound desc = could not find container \"a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8\": container with ID starting with a1ebf31f0c2d61ad890f1b8868ae4165f635b98535e3b3b4550f34029b5bd2c8 not found: ID does not exist"
Sep 09 20:49:35.341179 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.341157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33f95c14-6d03-478c-bc70-f62c73eb9b61-config-file\") pod \"33f95c14-6d03-478c-bc70-f62c73eb9b61\" (UID: \"33f95c14-6d03-478c-bc70-f62c73eb9b61\") "
Sep 09 20:49:35.341274 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.341187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rfzc4\" (UniqueName: \"kubernetes.io/projected/33f95c14-6d03-478c-bc70-f62c73eb9b61-kube-api-access-rfzc4\") pod \"33f95c14-6d03-478c-bc70-f62c73eb9b61\" (UID: \"33f95c14-6d03-478c-bc70-f62c73eb9b61\") "
Sep 09 20:49:35.341515 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.341496    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f95c14-6d03-478c-bc70-f62c73eb9b61-config-file" (OuterVolumeSpecName: "config-file") pod "33f95c14-6d03-478c-bc70-f62c73eb9b61" (UID: "33f95c14-6d03-478c-bc70-f62c73eb9b61"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:49:35.343180 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.343160    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f95c14-6d03-478c-bc70-f62c73eb9b61-kube-api-access-rfzc4" (OuterVolumeSpecName: "kube-api-access-rfzc4") pod "33f95c14-6d03-478c-bc70-f62c73eb9b61" (UID: "33f95c14-6d03-478c-bc70-f62c73eb9b61"). InnerVolumeSpecName "kube-api-access-rfzc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:49:35.441843 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.441778    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/33f95c14-6d03-478c-bc70-f62c73eb9b61-config-file\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:49:35.441843 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.441800    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rfzc4\" (UniqueName: \"kubernetes.io/projected/33f95c14-6d03-478c-bc70-f62c73eb9b61-kube-api-access-rfzc4\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:49:35.642410 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.642379    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mm4dg"]
Sep 09 20:49:35.647534 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:35.647512    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-mm4dg"]
Sep 09 20:49:36.285369 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:36.285339    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33f95c14-6d03-478c-bc70-f62c73eb9b61" path="/var/lib/kubelet/pods/33f95c14-6d03-478c-bc70-f62c73eb9b61/volumes"
Sep 09 20:49:37.337492 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:37.337460    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:37.433199 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:37.433161    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:37.814120 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:37.814089    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:50.738517 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:50.738482    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:51.146820 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:51.146734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:58.073174 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.073129    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:58.105946 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.105919    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-687cc689d8-7wjnr"]
Sep 09 20:49:58.106211 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.106200    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33f95c14-6d03-478c-bc70-f62c73eb9b61" containerName="limitador"
Sep 09 20:49:58.106261 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.106213    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="33f95c14-6d03-478c-bc70-f62c73eb9b61" containerName="limitador"
Sep 09 20:49:58.106295 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.106269    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="33f95c14-6d03-478c-bc70-f62c73eb9b61" containerName="limitador"
Sep 09 20:49:58.113634 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.113610    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.116485 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.116465    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mbhzx\""
Sep 09 20:49:58.116737 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.116721    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 20:49:58.116815 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.116722    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 20:49:58.117224 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.117210    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 20:49:58.120852 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.120819    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687cc689d8-7wjnr"]
Sep 09 20:49:58.156386 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.156350    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:58.213512 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.213486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gngb7\" (UniqueName: \"kubernetes.io/projected/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-kube-api-access-gngb7\") pod \"maas-controller-687cc689d8-7wjnr\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") " pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.213640 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.213520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-metrics-certs\") pod \"maas-controller-687cc689d8-7wjnr\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") " pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.213640 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.213619    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-webhook-cert\") pod \"maas-controller-687cc689d8-7wjnr\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") " pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.216798 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.216770    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:58.265716 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.265691    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-qzg8v"]
Sep 09 20:49:58.269004 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.268991    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.279006 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.278982    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-qzg8v"]
Sep 09 20:49:58.288670 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.288642    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:58.314982 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.314957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-webhook-cert\") pod \"maas-controller-687cc689d8-7wjnr\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") " pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.315076 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.315004    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gngb7\" (UniqueName: \"kubernetes.io/projected/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-kube-api-access-gngb7\") pod \"maas-controller-687cc689d8-7wjnr\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") " pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.315076 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.315024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-metrics-certs\") pod \"maas-controller-687cc689d8-7wjnr\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") " pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.317396 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.317370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-webhook-cert\") pod \"maas-controller-687cc689d8-7wjnr\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") " pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.317535 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.317516    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-metrics-certs\") pod \"maas-controller-687cc689d8-7wjnr\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") " pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.323944 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.323895    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gngb7\" (UniqueName: \"kubernetes.io/projected/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-kube-api-access-gngb7\") pod \"maas-controller-687cc689d8-7wjnr\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") " pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.353007 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.352975    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:49:58.415330 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.415299    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-webhook-cert\") pod \"maas-controller-86f6d44ffc-qzg8v\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") " pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.415414 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.415347    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzhqn\" (UniqueName: \"kubernetes.io/projected/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-kube-api-access-bzhqn\") pod \"maas-controller-86f6d44ffc-qzg8v\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") " pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.415498 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.415482    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-metrics-certs\") pod \"maas-controller-86f6d44ffc-qzg8v\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") " pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.424177 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.424158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:49:58.516714 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.516688    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-webhook-cert\") pod \"maas-controller-86f6d44ffc-qzg8v\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") " pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.516859 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.516725    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzhqn\" (UniqueName: \"kubernetes.io/projected/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-kube-api-access-bzhqn\") pod \"maas-controller-86f6d44ffc-qzg8v\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") " pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.516859 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.516753    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-metrics-certs\") pod \"maas-controller-86f6d44ffc-qzg8v\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") " pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.519026 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.518997    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-metrics-certs\") pod \"maas-controller-86f6d44ffc-qzg8v\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") " pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.519131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.519036    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-webhook-cert\") pod \"maas-controller-86f6d44ffc-qzg8v\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") " pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.524917 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.524892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzhqn\" (UniqueName: \"kubernetes.io/projected/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-kube-api-access-bzhqn\") pod \"maas-controller-86f6d44ffc-qzg8v\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") " pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.579181 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.579130    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:49:58.698697 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.698659    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-687cc689d8-7wjnr"]
Sep 09 20:49:58.701383 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:49:58.701358    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a2be05_d6fe_4f4b_bb2c_c599743b69d6.slice/crio-b8b9f34429c8385eef4ec4af1fc0f095380534cd4394fc6949809eb9b2baa105 WatchSource:0}: Error finding container b8b9f34429c8385eef4ec4af1fc0f095380534cd4394fc6949809eb9b2baa105: Status 404 returned error can't find the container with id b8b9f34429c8385eef4ec4af1fc0f095380534cd4394fc6949809eb9b2baa105
Sep 09 20:49:58.848067 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:58.848041    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-qzg8v"]
Sep 09 20:49:58.850088 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:49:58.850064    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7df9cab9_ee01_4b01_83a1_7d700b86c6aa.slice/crio-1dd7a96d1775258c194e6c601dfa84e333b4690fd021d027b37cb0c87bf872f8 WatchSource:0}: Error finding container 1dd7a96d1775258c194e6c601dfa84e333b4690fd021d027b37cb0c87bf872f8: Status 404 returned error can't find the container with id 1dd7a96d1775258c194e6c601dfa84e333b4690fd021d027b37cb0c87bf872f8
Sep 09 20:49:59.423354 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:59.423301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687cc689d8-7wjnr" event={"ID":"39a2be05-d6fe-4f4b-bb2c-c599743b69d6","Type":"ContainerStarted","Data":"b8b9f34429c8385eef4ec4af1fc0f095380534cd4394fc6949809eb9b2baa105"}
Sep 09 20:49:59.424581 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:49:59.424554    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v" event={"ID":"7df9cab9-ee01-4b01-83a1-7d700b86c6aa","Type":"ContainerStarted","Data":"1dd7a96d1775258c194e6c601dfa84e333b4690fd021d027b37cb0c87bf872f8"}
Sep 09 20:50:01.644376 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:01.644319    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:01.743498 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:01.743464    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:02.441465 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:02.441428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687cc689d8-7wjnr" event={"ID":"39a2be05-d6fe-4f4b-bb2c-c599743b69d6","Type":"ContainerStarted","Data":"98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d"}
Sep 09 20:50:02.441658 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:02.441493    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:50:02.442998 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:02.442973    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v" event={"ID":"7df9cab9-ee01-4b01-83a1-7d700b86c6aa","Type":"ContainerStarted","Data":"25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92"}
Sep 09 20:50:02.443137 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:02.443028    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:50:02.485517 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:02.485392    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-687cc689d8-7wjnr" podStartSLOduration=1.734536871 podStartE2EDuration="4.485373535s" totalImagesPullingTime="2.750836664s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:02.484172142 +0000 UTC m=+582.676009741" watchObservedRunningTime="2026-09-09 20:50:02.485373535 +0000 UTC m=+582.677211134"
Sep 09 20:50:02.515867 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:02.515821    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v" podStartSLOduration=1.907180801 podStartE2EDuration="4.515806735s" totalImagesPullingTime="2.608625934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:49:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:02.515062501 +0000 UTC m=+582.706900100" watchObservedRunningTime="2026-09-09 20:50:02.515806735 +0000 UTC m=+582.707644334"
Sep 09 20:50:13.449544 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.449510    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 20:50:13.450007 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.449599    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:50:13.515493 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.515460    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687cc689d8-7wjnr"]
Sep 09 20:50:13.515687 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.515643    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-687cc689d8-7wjnr" podUID="39a2be05-d6fe-4f4b-bb2c-c599743b69d6" containerName="manager" containerID="cri-o://98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d" gracePeriod=30
Sep 09 20:50:13.755224 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.755197    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:50:13.934366 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.934331    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-webhook-cert\") pod \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") "
Sep 09 20:50:13.934559 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.934407    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-metrics-certs\") pod \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") "
Sep 09 20:50:13.934559 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.934467    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gngb7\" (UniqueName: \"kubernetes.io/projected/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-kube-api-access-gngb7\") pod \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\" (UID: \"39a2be05-d6fe-4f4b-bb2c-c599743b69d6\") "
Sep 09 20:50:13.936536 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.936508    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-kube-api-access-gngb7" (OuterVolumeSpecName: "kube-api-access-gngb7") pod "39a2be05-d6fe-4f4b-bb2c-c599743b69d6" (UID: "39a2be05-d6fe-4f4b-bb2c-c599743b69d6"). InnerVolumeSpecName "kube-api-access-gngb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:50:13.936642 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.936542    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "39a2be05-d6fe-4f4b-bb2c-c599743b69d6" (UID: "39a2be05-d6fe-4f4b-bb2c-c599743b69d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:13.936642 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:13.936593    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "39a2be05-d6fe-4f4b-bb2c-c599743b69d6" (UID: "39a2be05-d6fe-4f4b-bb2c-c599743b69d6"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:50:14.035774 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.035694    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gngb7\" (UniqueName: \"kubernetes.io/projected/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-kube-api-access-gngb7\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:50:14.035774 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.035723    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-webhook-cert\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:50:14.035774 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.035733    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/39a2be05-d6fe-4f4b-bb2c-c599743b69d6-metrics-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:50:14.493830 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.493793    2566 generic.go:356] "Generic (PLEG): container finished" podID="39a2be05-d6fe-4f4b-bb2c-c599743b69d6" containerID="98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d" exitCode=0
Sep 09 20:50:14.493830 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.493832    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-687cc689d8-7wjnr"
Sep 09 20:50:14.494271 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.493865    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687cc689d8-7wjnr" event={"ID":"39a2be05-d6fe-4f4b-bb2c-c599743b69d6","Type":"ContainerDied","Data":"98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d"}
Sep 09 20:50:14.494271 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.493909    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-687cc689d8-7wjnr" event={"ID":"39a2be05-d6fe-4f4b-bb2c-c599743b69d6","Type":"ContainerDied","Data":"b8b9f34429c8385eef4ec4af1fc0f095380534cd4394fc6949809eb9b2baa105"}
Sep 09 20:50:14.494271 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.493933    2566 scope.go:117] "RemoveContainer" containerID="98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d"
Sep 09 20:50:14.502135 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.502119    2566 scope.go:117] "RemoveContainer" containerID="98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d"
Sep 09 20:50:14.502420 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:50:14.502401    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d\": container with ID starting with 98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d not found: ID does not exist" containerID="98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d"
Sep 09 20:50:14.502484 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.502431    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d"} err="failed to get container status \"98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d\": rpc error: code = NotFound desc = could not find container \"98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d\": container with ID starting with 98acf2fc47d7f7f0fac8f9d674b3b0f298614354788c51c1b419535acc44068d not found: ID does not exist"
Sep 09 20:50:14.510989 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.510968    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-687cc689d8-7wjnr"]
Sep 09 20:50:14.514248 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:14.514227    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-687cc689d8-7wjnr"]
Sep 09 20:50:16.286140 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:16.286109    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a2be05-d6fe-4f4b-bb2c-c599743b69d6" path="/var/lib/kubelet/pods/39a2be05-d6fe-4f4b-bb2c-c599743b69d6/volumes"
Sep 09 20:50:19.775216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:19.775178    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:30.324687 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:30.324649    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:30.492573 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:30.492534    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:31.735521 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.735477    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:31.839830 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.839798    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:31.881973 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.881939    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"]
Sep 09 20:50:31.882289 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.882273    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a2be05-d6fe-4f4b-bb2c-c599743b69d6" containerName="manager"
Sep 09 20:50:31.882398 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.882292    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a2be05-d6fe-4f4b-bb2c-c599743b69d6" containerName="manager"
Sep 09 20:50:31.882458 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.882411    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a2be05-d6fe-4f4b-bb2c-c599743b69d6" containerName="manager"
Sep 09 20:50:31.885172 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.885150    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:31.887672 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.887653    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 09 20:50:31.888523 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.888499    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 09 20:50:31.888523 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.888513    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 09 20:50:31.888656 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.888514    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-2mhvs\""
Sep 09 20:50:31.888656 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.888519    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 09 20:50:31.895577 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.895554    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"]
Sep 09 20:50:31.922001 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.921971    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:31.969913 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.969884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-openshift-service-ca\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:31.970045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.969928    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-metrics-certs\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:31.970045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.969953    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-maas-api-tls\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:31.970045 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:31.970023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxh2w\" (UniqueName: \"kubernetes.io/projected/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-kube-api-access-zxh2w\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.071357 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.071276    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-openshift-service-ca\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.071357 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.071345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-metrics-certs\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.071544 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.071370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-maas-api-tls\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.071544 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.071402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxh2w\" (UniqueName: \"kubernetes.io/projected/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-kube-api-access-zxh2w\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.071945 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.071923    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-openshift-service-ca\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.074003 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.073981    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-maas-api-tls\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.074108 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.074023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-metrics-certs\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.079695 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.079673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxh2w\" (UniqueName: \"kubernetes.io/projected/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-kube-api-access-zxh2w\") pod \"maas-api-6ffbd7f869-tl22k\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") " pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.164487 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.164446    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:32.195453 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.195427    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:32.267217 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.267181    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:32.375424 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.375302    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:32.379841 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.379815    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"]
Sep 09 20:50:32.383155 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.383138    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:32.385536 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.385515    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-2ln58\""
Sep 09 20:50:32.385815 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.385800    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 09 20:50:32.394405 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.394385    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"]
Sep 09 20:50:32.470851 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.470822    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"]
Sep 09 20:50:32.473787 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:50:32.473760    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebbac2d_8a2b_4cdc_af8f_fc8b23101a56.slice/crio-e4eed5c52dda5a71c1aef4f7954360bdaf13118655d64943aeebb7d1965598da WatchSource:0}: Error finding container e4eed5c52dda5a71c1aef4f7954360bdaf13118655d64943aeebb7d1965598da: Status 404 returned error can't find the container with id e4eed5c52dda5a71c1aef4f7954360bdaf13118655d64943aeebb7d1965598da
Sep 09 20:50:32.473787 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.473775    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecfe544-adc8-4087-bc4e-afd45adf8f00-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dhhh5\" (UID: \"eecfe544-adc8-4087-bc4e-afd45adf8f00\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:32.473932 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.473859    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfwwx\" (UniqueName: \"kubernetes.io/projected/eecfe544-adc8-4087-bc4e-afd45adf8f00-kube-api-access-pfwwx\") pod \"payload-pre-processing-59fbc47bf8-dhhh5\" (UID: \"eecfe544-adc8-4087-bc4e-afd45adf8f00\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:32.477425 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.477408    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:50:32.570770 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.570735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k" event={"ID":"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56","Type":"ContainerStarted","Data":"e4eed5c52dda5a71c1aef4f7954360bdaf13118655d64943aeebb7d1965598da"}
Sep 09 20:50:32.575009 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.574988    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfwwx\" (UniqueName: \"kubernetes.io/projected/eecfe544-adc8-4087-bc4e-afd45adf8f00-kube-api-access-pfwwx\") pod \"payload-pre-processing-59fbc47bf8-dhhh5\" (UID: \"eecfe544-adc8-4087-bc4e-afd45adf8f00\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:32.575105 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.575062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecfe544-adc8-4087-bc4e-afd45adf8f00-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dhhh5\" (UID: \"eecfe544-adc8-4087-bc4e-afd45adf8f00\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:32.575912 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.575889    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eecfe544-adc8-4087-bc4e-afd45adf8f00-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-dhhh5\" (UID: \"eecfe544-adc8-4087-bc4e-afd45adf8f00\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:32.584888 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.584862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfwwx\" (UniqueName: \"kubernetes.io/projected/eecfe544-adc8-4087-bc4e-afd45adf8f00-kube-api-access-pfwwx\") pod \"payload-pre-processing-59fbc47bf8-dhhh5\" (UID: \"eecfe544-adc8-4087-bc4e-afd45adf8f00\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:32.694348 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.694236    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:32.967060 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:32.966923    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"]
Sep 09 20:50:32.972323 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:50:32.972282    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecfe544_adc8_4087_bc4e_afd45adf8f00.slice/crio-e5164c031e21da532b949f43e2a790cf09bbe368bb8304afb2650ceca3185fe7 WatchSource:0}: Error finding container e5164c031e21da532b949f43e2a790cf09bbe368bb8304afb2650ceca3185fe7: Status 404 returned error can't find the container with id e5164c031e21da532b949f43e2a790cf09bbe368bb8304afb2650ceca3185fe7
Sep 09 20:50:33.577246 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:33.577207    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5" event={"ID":"eecfe544-adc8-4087-bc4e-afd45adf8f00","Type":"ContainerStarted","Data":"e5164c031e21da532b949f43e2a790cf09bbe368bb8304afb2650ceca3185fe7"}
Sep 09 20:50:35.589839 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:35.589800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k" event={"ID":"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56","Type":"ContainerStarted","Data":"a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134"}
Sep 09 20:50:35.590269 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:35.589932    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:35.591274 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:35.591253    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5" event={"ID":"eecfe544-adc8-4087-bc4e-afd45adf8f00","Type":"ContainerStarted","Data":"85632b2535845d40c0477e1a5e4e5440d27812e0662ee23d2bb073a28b70ddd4"}
Sep 09 20:50:35.591365 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:35.591330    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:35.613541 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:35.613493    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k" podStartSLOduration=2.008549702 podStartE2EDuration="4.613477683s" totalImagesPullingTime="2.604927981s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:35.612151328 +0000 UTC m=+615.803988928" watchObservedRunningTime="2026-09-09 20:50:35.613477683 +0000 UTC m=+615.805315282"
Sep 09 20:50:41.600317 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:41.600240    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 20:50:41.617925 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:41.617894    2566 generic.go:356] "Generic (PLEG): container finished" podID="712d1861-3ac0-4548-9336-953dc2d2af00" containerID="e79ffdd33f2e51b242978a40311843f08661c84a57c151f9cb28b171d2d1ead2" exitCode=0
Sep 09 20:50:41.618049 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:41.617969    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" event={"ID":"712d1861-3ac0-4548-9336-953dc2d2af00","Type":"ContainerDied","Data":"e79ffdd33f2e51b242978a40311843f08661c84a57c151f9cb28b171d2d1ead2"}
Sep 09 20:50:41.618461 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:41.618433    2566 scope.go:117] "RemoveContainer" containerID="e79ffdd33f2e51b242978a40311843f08661c84a57c151f9cb28b171d2d1ead2"
Sep 09 20:50:41.619728 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:41.619259    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5" podStartSLOduration=7.490846375 podStartE2EDuration="9.619246098s" totalImagesPullingTime="2.128399723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:50:35.62965529 +0000 UTC m=+615.821492890" watchObservedRunningTime="2026-09-09 20:50:41.619246098 +0000 UTC m=+621.811083697"
Sep 09 20:50:41.619728 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:41.619653    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:50:41.619826 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:41.619726    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:50:41.619826 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:41.619760    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:50:42.576632 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:42.576514    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:42.627270 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:42.627234    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" event={"ID":"712d1861-3ac0-4548-9336-953dc2d2af00","Type":"ContainerStarted","Data":"63e2247000236f30790a7b13c5edc3727c78bf942d83525195566ff9f5954b94"}
Sep 09 20:50:42.768752 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:42.768713    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:50:42.769021 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:42.768997    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7mxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:50:42.769123 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:42.769054    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" podUID="712d1861-3ac0-4548-9336-953dc2d2af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 09 20:50:43.769081 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:43.769045    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7mxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:50:43.769583 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:43.769101    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" podUID="712d1861-3ac0-4548-9336-953dc2d2af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 09 20:50:44.220033 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:44.219941    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:44.769195 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:44.769160    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7mxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:50:44.769609 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:44.769228    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" podUID="712d1861-3ac0-4548-9336-953dc2d2af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 09 20:50:45.769429 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:45.769392    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7mxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:50:45.769819 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:45.769451    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" podUID="712d1861-3ac0-4548-9336-953dc2d2af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 09 20:50:46.597462 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:46.597433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-dhhh5"
Sep 09 20:50:46.768554 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:46.768517    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-7mxfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused" start-of-body=
Sep 09 20:50:46.768751 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:46.768582    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq" podUID="712d1861-3ac0-4548-9336-953dc2d2af00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.26:15021/healthz/ready\": dial tcp 10.132.0.26:15021: connect: connection refused"
Sep 09 20:50:47.772213 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:47.772184    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:50:47.772578 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:47.772564    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:50:47.773094 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:47.773075    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-7mxfq"
Sep 09 20:50:48.740563 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:48.740529    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:58.270448 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.270417    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"]
Sep 09 20:50:58.273602 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.273581    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.276020 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.275996    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 09 20:50:58.276139 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.276037    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 09 20:50:58.276139 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.276084    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-szvm2\""
Sep 09 20:50:58.277018 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.277003    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 09 20:50:58.287849 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.287828    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"]
Sep 09 20:50:58.320388 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.320357    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:50:58.370607 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.370581    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.370727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.370620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.370727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.370640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.370727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.370664    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.370727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.370687    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnjfm\" (UniqueName: \"kubernetes.io/projected/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-kube-api-access-qnjfm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.370727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.370716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.471724 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.471691    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.471897 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.471749    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.471897 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.471785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.471897 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.471812    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.471897 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.471838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnjfm\" (UniqueName: \"kubernetes.io/projected/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-kube-api-access-qnjfm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.471897 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.471873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.472187 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.472160    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.472262 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.472229    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.472505 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.472485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.474081 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.474056    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.474366 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.474347    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.480195 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.480170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnjfm\" (UniqueName: \"kubernetes.io/projected/3b325e10-690d-4bfb-8bfa-5e3f113fa9f7-kube-api-access-qnjfm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz\" (UID: \"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.585833 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.585776    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:50:58.668170 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.668141    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:50:58.863644 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.863571    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"]
Sep 09 20:50:58.866418 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:50:58.866384    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b325e10_690d_4bfb_8bfa_5e3f113fa9f7.slice/crio-6fb550327179f33fcf30c2f86c98da80629b41bcfa6d22bd2ceed97d6abc780c WatchSource:0}: Error finding container 6fb550327179f33fcf30c2f86c98da80629b41bcfa6d22bd2ceed97d6abc780c: Status 404 returned error can't find the container with id 6fb550327179f33fcf30c2f86c98da80629b41bcfa6d22bd2ceed97d6abc780c
Sep 09 20:50:58.868329 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:58.868286    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 20:50:59.716826 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:50:59.716788    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz" event={"ID":"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7","Type":"ContainerStarted","Data":"6fb550327179f33fcf30c2f86c98da80629b41bcfa6d22bd2ceed97d6abc780c"}
Sep 09 20:51:00.722053 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:00.722013    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz" event={"ID":"3b325e10-690d-4bfb-8bfa-5e3f113fa9f7","Type":"ContainerStarted","Data":"8f66acd899e37afc748f750ad837ab3af8fe88921199c1cc4e7515ee11844729"}
Sep 09 20:51:00.745319 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:00.745253    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz" podStartSLOduration=1.682460852 podStartE2EDuration="2.745238636s" totalImagesPullingTime="1.062777784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:00.741535603 +0000 UTC m=+640.933373202" watchObservedRunningTime="2026-09-09 20:51:00.745238636 +0000 UTC m=+640.937076277"
Sep 09 20:51:03.352131 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:03.352091    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:08.586061 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:08.586027    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:51:08.586457 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:08.586087    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:51:08.598896 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:08.598868    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:51:08.767031 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:08.767000    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz"
Sep 09 20:51:19.697214 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:19.697168    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:20.467518 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:20.467487    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:51:25.928683 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:25.928642    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:26.858949 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:26.858915    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:51:37.469830 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.469790    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"]
Sep 09 20:51:37.475568 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.475546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.478167 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.478145    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 09 20:51:37.485595 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.485574    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"]
Sep 09 20:51:37.502213 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.502190    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:37.669127 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.669096    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.669271 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.669144    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vcw5b\" (UniqueName: \"kubernetes.io/projected/5f767224-76fc-4652-b57e-1110eb4d8808-kube-api-access-vcw5b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.669271 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.669219    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.669271 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.669260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.669402 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.669287    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.669402 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.669338    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f767224-76fc-4652-b57e-1110eb4d8808-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.769761 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.769735    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.769868 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.769772    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.769868 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.769794    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.769998 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.769976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f767224-76fc-4652-b57e-1110eb4d8808-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.770046 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.770024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.770111 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.770094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vcw5b\" (UniqueName: \"kubernetes.io/projected/5f767224-76fc-4652-b57e-1110eb4d8808-kube-api-access-vcw5b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.770162 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.770098    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.770213 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.770197    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.770442 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.770417    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.772040 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.772010    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f767224-76fc-4652-b57e-1110eb4d8808-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.772327 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.772296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f767224-76fc-4652-b57e-1110eb4d8808-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.781369 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.780773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vcw5b\" (UniqueName: \"kubernetes.io/projected/5f767224-76fc-4652-b57e-1110eb4d8808-kube-api-access-vcw5b\") pod \"e2e-trlp-test-simulated-kserve-598466548d-tc2xg\" (UID: \"5f767224-76fc-4652-b57e-1110eb4d8808\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:37.786205 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:37.786178    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:38.058679 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:38.058654    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"]
Sep 09 20:51:38.061115 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:51:38.061085    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f767224_76fc_4652_b57e_1110eb4d8808.slice/crio-b5fcb6d77ca644f24c7151df221d26318cd9a4f4d44da8290d74d8ba0c6967aa WatchSource:0}: Error finding container b5fcb6d77ca644f24c7151df221d26318cd9a4f4d44da8290d74d8ba0c6967aa: Status 404 returned error can't find the container with id b5fcb6d77ca644f24c7151df221d26318cd9a4f4d44da8290d74d8ba0c6967aa
Sep 09 20:51:38.156986 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:38.156950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:51:38.248208 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:38.248179    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:38.884387 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:38.884349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg" event={"ID":"5f767224-76fc-4652-b57e-1110eb4d8808","Type":"ContainerStarted","Data":"08f2870aa90a537bde86c23370e7ef8d6feca6332be0de3a18b5cabc651a5b24"}
Sep 09 20:51:38.884764 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:38.884402    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg" event={"ID":"5f767224-76fc-4652-b57e-1110eb4d8808","Type":"ContainerStarted","Data":"b5fcb6d77ca644f24c7151df221d26318cd9a4f4d44da8290d74d8ba0c6967aa"}
Sep 09 20:51:38.920433 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:38.920372    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg" podStartSLOduration=1.751301537 podStartE2EDuration="1.920353569s" totalImagesPullingTime="169.052032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:51:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:51:38.91970896 +0000 UTC m=+679.111546559" watchObservedRunningTime="2026-09-09 20:51:38.920353569 +0000 UTC m=+679.112191168"
Sep 09 20:51:47.786968 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:47.786922    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:47.787387 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:47.787046    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:47.799083 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:47.799059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:47.934429 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:47.934403    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-tc2xg"
Sep 09 20:51:50.429984 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:50.429951    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:51.965467 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:51.965435    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:51:53.209358 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:53.209328    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:51:59.633713 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:51:59.633675    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:01.676486 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.676445    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:52:01.688130 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.688102    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"]
Sep 09 20:52:01.691492 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.691472    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.693814 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.693788    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 09 20:52:01.704804 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.704777    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"]
Sep 09 20:52:01.736063 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.736031    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:01.755590 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.755565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.755730 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.755595    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.755730 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.755620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x69dl\" (UniqueName: \"kubernetes.io/projected/cb25a91f-f0ad-40db-8a12-22126a906aa1-kube-api-access-x69dl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.755730 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.755721    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb25a91f-f0ad-40db-8a12-22126a906aa1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.755863 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.755756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.755863 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.755781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.856735 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.856702    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb25a91f-f0ad-40db-8a12-22126a906aa1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.856921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.856742    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.856921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.856765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.856921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.856807    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.856921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.856826    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.856921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.856851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x69dl\" (UniqueName: \"kubernetes.io/projected/cb25a91f-f0ad-40db-8a12-22126a906aa1-kube-api-access-x69dl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.857199 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.857181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.857247 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.857227    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.857291 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.857272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.858997 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.858969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cb25a91f-f0ad-40db-8a12-22126a906aa1-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.859359 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.859340    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cb25a91f-f0ad-40db-8a12-22126a906aa1-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:01.868022 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:01.867999    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x69dl\" (UniqueName: \"kubernetes.io/projected/cb25a91f-f0ad-40db-8a12-22126a906aa1-kube-api-access-x69dl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-cvp25\" (UID: \"cb25a91f-f0ad-40db-8a12-22126a906aa1\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:02.000461 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:02.000428    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:02.272107 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:02.271975    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"]
Sep 09 20:52:02.274801 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:52:02.274775    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcb25a91f_f0ad_40db_8a12_22126a906aa1.slice/crio-11e221184059f51e1b40cbd76d8fd685e008c8e3480c034ba885b69deb45ca46 WatchSource:0}: Error finding container 11e221184059f51e1b40cbd76d8fd685e008c8e3480c034ba885b69deb45ca46: Status 404 returned error can't find the container with id 11e221184059f51e1b40cbd76d8fd685e008c8e3480c034ba885b69deb45ca46
Sep 09 20:52:02.992949 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:02.992914    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25" event={"ID":"cb25a91f-f0ad-40db-8a12-22126a906aa1","Type":"ContainerStarted","Data":"04cd2774efe3cd90f6a69f3f1cfc22132d59e9752fbb0de0b35731c7ca8b661c"}
Sep 09 20:52:02.993427 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:02.992959    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25" event={"ID":"cb25a91f-f0ad-40db-8a12-22126a906aa1","Type":"ContainerStarted","Data":"11e221184059f51e1b40cbd76d8fd685e008c8e3480c034ba885b69deb45ca46"}
Sep 09 20:52:03.011276 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:03.011232    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25" podStartSLOduration=1.82341262 podStartE2EDuration="2.011216014s" totalImagesPullingTime="187.803394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:52:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:52:03.009424806 +0000 UTC m=+703.201262405" watchObservedRunningTime="2026-09-09 20:52:03.011216014 +0000 UTC m=+703.203053613"
Sep 09 20:52:09.470467 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:09.470430    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:52:12.001109 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:12.001070    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:12.001109 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:12.001116    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:12.013624 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:12.013602    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:12.045362 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:12.045338    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-cvp25"
Sep 09 20:52:13.362590 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:13.362556    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:13.462834 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:13.462808    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:35.193151 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:35.193086    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:52:35.294792 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:52:35.294761    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:32.339682 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:32.339640    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:32.502781 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:32.502744    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:33.580260 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:33.580219    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:33.864806 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:33.864726    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:50.181329 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.181223    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:50.293351 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.293295    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:50.436484 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.436404    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:50.535005 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.534971    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"]
Sep 09 20:53:50.538128 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.538110    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.542646 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.542627    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-5eef90-openshift-default-dockercfg-tst98\""
Sep 09 20:53:50.560111 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.560087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"]
Sep 09 20:53:50.565568 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.565533    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:50.625879 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.625852    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istiod-ca-cert\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.626030 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.625886    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-certs\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.626030 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.625913    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-socket\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.626030 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.625958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-token\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.626175 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.626034    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-data\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.626175 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.626084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fgpq\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-kube-api-access-9fgpq\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.626278 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.626208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-podinfo\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.626345 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.626298    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-envoy\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.626406 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.626356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-credential-socket\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.686694 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.686626    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:50.727740 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.727711    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-credential-socket\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.727740 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.727744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istiod-ca-cert\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.727951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.727762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-certs\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.727951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.727778    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-socket\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.727951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.727800    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-token\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.727951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.727824    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-data\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.727951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.727853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fgpq\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-kube-api-access-9fgpq\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.727951 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.727905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-podinfo\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.728237 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.727974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-envoy\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.728237 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.728154    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-credential-socket\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.728373 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.728278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-certs\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.728425 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.728378    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-socket\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.728425 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.728410    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-data\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.728659 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.728638    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istiod-ca-cert\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.730872 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.730681    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-envoy\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.730965 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.730883    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-podinfo\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.737740 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.737717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-token\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.738970 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.738953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fgpq\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-kube-api-access-9fgpq\") pod \"e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") " pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.790571 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.790526    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:50.847852 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.847821    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:50.908234 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:50.908204    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:51.036316 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.036283    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:51.132123 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.132095    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"]
Sep 09 20:53:51.134370 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:53:51.134341    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffa8e0d1_e6d1_4678_b4c6_c99a96981487.slice/crio-cdfec216f2e6ff67a018300c9430f76afa477087b8afb2db194f64342f755c42 WatchSource:0}: Error finding container cdfec216f2e6ff67a018300c9430f76afa477087b8afb2db194f64342f755c42: Status 404 returned error can't find the container with id cdfec216f2e6ff67a018300c9430f76afa477087b8afb2db194f64342f755c42
Sep 09 20:53:51.136670 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.136636    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:53:51.136742 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.136725    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:53:51.136786 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.136767    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:53:51.454359 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.454251    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" event={"ID":"ffa8e0d1-e6d1-4678-b4c6-c99a96981487","Type":"ContainerStarted","Data":"9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6"}
Sep 09 20:53:51.454359 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.454327    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" event={"ID":"ffa8e0d1-e6d1-4678-b4c6-c99a96981487","Type":"ContainerStarted","Data":"cdfec216f2e6ff67a018300c9430f76afa477087b8afb2db194f64342f755c42"}
Sep 09 20:53:51.490891 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.490833    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" podStartSLOduration=1.4908117650000001 podStartE2EDuration="1.490811765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:51.489464157 +0000 UTC m=+811.681301759" watchObservedRunningTime="2026-09-09 20:53:51.490811765 +0000 UTC m=+811.682649365"
Sep 09 20:53:51.735070 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.735035    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:51.830328 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.830276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:51.848180 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:51.848150    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:52.225415 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:52.225336    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:52.853363 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:52.853329    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:53.468164 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:53.464411    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:53.469219 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:53.469193    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:53:53.509636 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:53.509601    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:53.693824 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:53.693791    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:53.874193 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:53.874157    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:54.017423 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:54.017388    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:54.337916 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:54.337884    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:53:54.866780 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:54.866743    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:55.092264 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:55.092230    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:55.886859 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:55.886823    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:56.317991 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.317955    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:56.579168 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.579082    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"]
Sep 09 20:53:56.582699 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.582676    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:53:56.587913 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.587893    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5eef90\""
Sep 09 20:53:56.606553 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.606523    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"]
Sep 09 20:53:56.675690 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.675657    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b35e8-81a5-4083-9ed9-87b218e01fc2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl\" (UID: \"645b35e8-81a5-4083-9ed9-87b218e01fc2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:53:56.675840 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.675696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km8nl\" (UniqueName: \"kubernetes.io/projected/645b35e8-81a5-4083-9ed9-87b218e01fc2-kube-api-access-km8nl\") pod \"payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl\" (UID: \"645b35e8-81a5-4083-9ed9-87b218e01fc2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:53:56.686195 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.686167    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"]
Sep 09 20:53:56.689259 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.689244    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:53:56.697442 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.697425    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5eef90-dockercfg-8kgff\""
Sep 09 20:53:56.709977 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.709948    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"]
Sep 09 20:53:56.735282 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.735252    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:56.776432 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.776405    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b35e8-81a5-4083-9ed9-87b218e01fc2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl\" (UID: \"645b35e8-81a5-4083-9ed9-87b218e01fc2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:53:56.776576 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.776442    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km8nl\" (UniqueName: \"kubernetes.io/projected/645b35e8-81a5-4083-9ed9-87b218e01fc2-kube-api-access-km8nl\") pod \"payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl\" (UID: \"645b35e8-81a5-4083-9ed9-87b218e01fc2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:53:56.776576 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.776472    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk\" (UID: \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:53:56.776576 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.776502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw6rk\" (UniqueName: \"kubernetes.io/projected/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-kube-api-access-jw6rk\") pod \"payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk\" (UID: \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:53:56.777234 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.777213    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b35e8-81a5-4083-9ed9-87b218e01fc2-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl\" (UID: \"645b35e8-81a5-4083-9ed9-87b218e01fc2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:53:56.798582 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.798556    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km8nl\" (UniqueName: \"kubernetes.io/projected/645b35e8-81a5-4083-9ed9-87b218e01fc2-kube-api-access-km8nl\") pod \"payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl\" (UID: \"645b35e8-81a5-4083-9ed9-87b218e01fc2\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:53:56.877930 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.877848    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk\" (UID: \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:53:56.877930 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.877885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw6rk\" (UniqueName: \"kubernetes.io/projected/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-kube-api-access-jw6rk\") pod \"payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk\" (UID: \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:53:56.878557 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.878537    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk\" (UID: \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:53:56.892834 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.892807    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:53:56.905429 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.905405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw6rk\" (UniqueName: \"kubernetes.io/projected/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-kube-api-access-jw6rk\") pod \"payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk\" (UID: \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:53:56.999700 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:56.999667    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:53:57.230364 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.230338    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"]
Sep 09 20:53:57.232209 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:53:57.232183    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645b35e8_81a5_4083_9ed9_87b218e01fc2.slice/crio-61236b30b21cecf45b9928dcdca1f097b9a09cf5156385cdda18db89e1b69acb WatchSource:0}: Error finding container 61236b30b21cecf45b9928dcdca1f097b9a09cf5156385cdda18db89e1b69acb: Status 404 returned error can't find the container with id 61236b30b21cecf45b9928dcdca1f097b9a09cf5156385cdda18db89e1b69acb
Sep 09 20:53:57.327806 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.327778    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"]
Sep 09 20:53:57.330348 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:53:57.330295    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32f1dac_f4ea_4c7b_a824_65316d76dfb6.slice/crio-2cfabaf4cdca49fea12ebdfe7a269220658bc3166cea928cb252f11e2b338e48 WatchSource:0}: Error finding container 2cfabaf4cdca49fea12ebdfe7a269220658bc3166cea928cb252f11e2b338e48: Status 404 returned error can't find the container with id 2cfabaf4cdca49fea12ebdfe7a269220658bc3166cea928cb252f11e2b338e48
Sep 09 20:53:57.481735 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.481648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk" event={"ID":"d32f1dac-f4ea-4c7b-a824-65316d76dfb6","Type":"ContainerStarted","Data":"93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457"}
Sep 09 20:53:57.481735 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.481722    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk" event={"ID":"d32f1dac-f4ea-4c7b-a824-65316d76dfb6","Type":"ContainerStarted","Data":"2cfabaf4cdca49fea12ebdfe7a269220658bc3166cea928cb252f11e2b338e48"}
Sep 09 20:53:57.481918 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.481759    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:53:57.483297 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.483273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl" event={"ID":"645b35e8-81a5-4083-9ed9-87b218e01fc2","Type":"ContainerStarted","Data":"e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37"}
Sep 09 20:53:57.483426 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.483362    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:53:57.483426 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.483389    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl" event={"ID":"645b35e8-81a5-4083-9ed9-87b218e01fc2","Type":"ContainerStarted","Data":"61236b30b21cecf45b9928dcdca1f097b9a09cf5156385cdda18db89e1b69acb"}
Sep 09 20:53:57.488450 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.488419    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"]
Sep 09 20:53:57.491550 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.491534    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.496895 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.496875    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-47e41c-serving-cert\""
Sep 09 20:53:57.531391 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.531367    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"]
Sep 09 20:53:57.550977 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.550948    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:57.584265 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.584232    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2b5add-ae0d-425f-8f97-256709aa7865-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.584432 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.584391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-maas-api-tls\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.584501 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.584426    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm4nr\" (UniqueName: \"kubernetes.io/projected/4d2b5add-ae0d-425f-8f97-256709aa7865-kube-api-access-wm4nr\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.584559 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.584520    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-metrics-certs\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.606369 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.605087    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk" podStartSLOduration=1.605061318 podStartE2EDuration="1.605061318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:57.549977701 +0000 UTC m=+817.741815301" watchObservedRunningTime="2026-09-09 20:53:57.605061318 +0000 UTC m=+817.796898922"
Sep 09 20:53:57.642394 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.642350    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl" podStartSLOduration=1.642335604 podStartE2EDuration="1.642335604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:57.606689868 +0000 UTC m=+817.798527468" watchObservedRunningTime="2026-09-09 20:53:57.642335604 +0000 UTC m=+817.834173202"
Sep 09 20:53:57.676012 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.675975    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:57.685127 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.685105    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-metrics-certs\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.685224 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.685154    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2b5add-ae0d-425f-8f97-256709aa7865-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.685224 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.685186    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-maas-api-tls\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.685224 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.685201    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm4nr\" (UniqueName: \"kubernetes.io/projected/4d2b5add-ae0d-425f-8f97-256709aa7865-kube-api-access-wm4nr\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.685407 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:53:57.685346    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-serving-cert: secret "maas-api-e2e-worker-w4-47e41c-serving-cert" not found
Sep 09 20:53:57.685464 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:53:57.685408    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-maas-api-tls podName:4d2b5add-ae0d-425f-8f97-256709aa7865 nodeName:}" failed. No retries permitted until 2026-09-09 20:53:58.185391444 +0000 UTC m=+818.377229037 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-maas-api-tls") pod "maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb" (UID: "4d2b5add-ae0d-425f-8f97-256709aa7865") : secret "maas-api-e2e-worker-w4-47e41c-serving-cert" not found
Sep 09 20:53:57.685904 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.685885    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2b5add-ae0d-425f-8f97-256709aa7865-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.687615 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.687599    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-metrics-certs\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.700799 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.700778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm4nr\" (UniqueName: \"kubernetes.io/projected/4d2b5add-ae0d-425f-8f97-256709aa7865-kube-api-access-wm4nr\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:57.964236 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:57.964154    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:58.190917 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:58.190883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-maas-api-tls\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:58.193414 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:58.193381    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-maas-api-tls\") pod \"maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:58.401668 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:58.401627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:58.532436 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:58.532393    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:58.720684 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:53:58.720657    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d2b5add_ae0d_425f_8f97_256709aa7865.slice/crio-c2436fc55d9842375fd260f347417690a281c39df12347d3d7c92a4742660b25 WatchSource:0}: Error finding container c2436fc55d9842375fd260f347417690a281c39df12347d3d7c92a4742660b25: Status 404 returned error can't find the container with id c2436fc55d9842375fd260f347417690a281c39df12347d3d7c92a4742660b25
Sep 09 20:53:58.724456 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:58.724435    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"]
Sep 09 20:53:58.727078 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:58.727049    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:59.158353 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.158317    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"]
Sep 09 20:53:59.162785 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.162768    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:53:59.166908 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.166882    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-47e41c\""
Sep 09 20:53:59.167161 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.167144    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-47e41c-dockercfg-zdcbf\""
Sep 09 20:53:59.175276 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.175253    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"]
Sep 09 20:53:59.199609 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.199589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ddfb64-e9be-417a-91cf-fc6257a76f95-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl\" (UID: \"82ddfb64-e9be-417a-91cf-fc6257a76f95\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:53:59.199699 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.199649    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7d2z\" (UniqueName: \"kubernetes.io/projected/82ddfb64-e9be-417a-91cf-fc6257a76f95-kube-api-access-b7d2z\") pod \"payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl\" (UID: \"82ddfb64-e9be-417a-91cf-fc6257a76f95\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:53:59.300609 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.300577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ddfb64-e9be-417a-91cf-fc6257a76f95-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl\" (UID: \"82ddfb64-e9be-417a-91cf-fc6257a76f95\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:53:59.300722 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.300671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7d2z\" (UniqueName: \"kubernetes.io/projected/82ddfb64-e9be-417a-91cf-fc6257a76f95-kube-api-access-b7d2z\") pod \"payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl\" (UID: \"82ddfb64-e9be-417a-91cf-fc6257a76f95\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:53:59.301378 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.301358    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ddfb64-e9be-417a-91cf-fc6257a76f95-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl\" (UID: \"82ddfb64-e9be-417a-91cf-fc6257a76f95\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:53:59.315832 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.315802    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7d2z\" (UniqueName: \"kubernetes.io/projected/82ddfb64-e9be-417a-91cf-fc6257a76f95-kube-api-access-b7d2z\") pod \"payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl\" (UID: \"82ddfb64-e9be-417a-91cf-fc6257a76f95\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:53:59.455444 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.455414    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:59.473281 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.473257    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:53:59.496147 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.496124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb" event={"ID":"4d2b5add-ae0d-425f-8f97-256709aa7865","Type":"ContainerStarted","Data":"bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54"}
Sep 09 20:53:59.496241 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.496169    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb" event={"ID":"4d2b5add-ae0d-425f-8f97-256709aa7865","Type":"ContainerStarted","Data":"c2436fc55d9842375fd260f347417690a281c39df12347d3d7c92a4742660b25"}
Sep 09 20:53:59.496241 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.496206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:53:59.545965 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.545904    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb" podStartSLOduration=1.8832177140000002 podStartE2EDuration="2.545884175s" totalImagesPullingTime="662.666461ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:53:59.54356919 +0000 UTC m=+819.735406789" watchObservedRunningTime="2026-09-09 20:53:59.545884175 +0000 UTC m=+819.737721777"
Sep 09 20:53:59.595820 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.595787    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:53:59.759684 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:53:59.759653    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"]
Sep 09 20:53:59.762802 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:53:59.762776    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ddfb64_e9be_417a_91cf_fc6257a76f95.slice/crio-0706d0c25618458d3f5be5518e778418cdbb29381d3c91faae896c5dd5e99f44 WatchSource:0}: Error finding container 0706d0c25618458d3f5be5518e778418cdbb29381d3c91faae896c5dd5e99f44: Status 404 returned error can't find the container with id 0706d0c25618458d3f5be5518e778418cdbb29381d3c91faae896c5dd5e99f44
Sep 09 20:54:00.100638 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.100545    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:00.248668 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.248634    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:00.349593 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.349557    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:00.502087 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.502057    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl" event={"ID":"82ddfb64-e9be-417a-91cf-fc6257a76f95","Type":"ContainerStarted","Data":"ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b"}
Sep 09 20:54:00.502499 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.502115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl" event={"ID":"82ddfb64-e9be-417a-91cf-fc6257a76f95","Type":"ContainerStarted","Data":"0706d0c25618458d3f5be5518e778418cdbb29381d3c91faae896c5dd5e99f44"}
Sep 09 20:54:00.502499 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.502227    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:54:00.539142 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.539111    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"]
Sep 09 20:54:00.542603 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.542584    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:00.545390 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.545368    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-d36b91-dockercfg-jtk9p\""
Sep 09 20:54:00.545498 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.545460    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-d36b91\""
Sep 09 20:54:00.571910 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.571869    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl" podStartSLOduration=1.5718560080000001 podStartE2EDuration="1.571856008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:53:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:00.545375861 +0000 UTC m=+820.737213461" watchObservedRunningTime="2026-09-09 20:54:00.571856008 +0000 UTC m=+820.763693607"
Sep 09 20:54:00.573814 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.573793    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"]
Sep 09 20:54:00.613033 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.613008    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87m6\" (UniqueName: \"kubernetes.io/projected/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-kube-api-access-b87m6\") pod \"payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf\" (UID: \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:00.613358 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.613328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf\" (UID: \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:00.714583 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.714548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf\" (UID: \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:00.714740 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.714601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87m6\" (UniqueName: \"kubernetes.io/projected/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-kube-api-access-b87m6\") pod \"payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf\" (UID: \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:00.715346 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.715304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf\" (UID: \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:00.738164 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.738136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87m6\" (UniqueName: \"kubernetes.io/projected/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-kube-api-access-b87m6\") pod \"payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf\" (UID: \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:00.853571 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.853492    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:00.890866 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:00.890837    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:01.143391 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:01.143359    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"]
Sep 09 20:54:01.143522 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:54:01.143404    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eb83949_b8fa_46d5_8fec_3d1a8b823b28.slice/crio-4f958f42ef4f4b819b0795b610a11bd4796628a7aa717ba0e7ffc9245882db6e WatchSource:0}: Error finding container 4f958f42ef4f4b819b0795b610a11bd4796628a7aa717ba0e7ffc9245882db6e: Status 404 returned error can't find the container with id 4f958f42ef4f4b819b0795b610a11bd4796628a7aa717ba0e7ffc9245882db6e
Sep 09 20:54:01.159850 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:01.159743    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:01.397163 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:01.397057    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:01.508092 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:01.508058    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf" event={"ID":"4eb83949-b8fa-46d5-8fec-3d1a8b823b28","Type":"ContainerStarted","Data":"0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d"}
Sep 09 20:54:01.508536 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:01.508183    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:01.508536 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:01.508204    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf" event={"ID":"4eb83949-b8fa-46d5-8fec-3d1a8b823b28","Type":"ContainerStarted","Data":"4f958f42ef4f4b819b0795b610a11bd4796628a7aa717ba0e7ffc9245882db6e"}
Sep 09 20:54:01.556262 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:01.556215    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf" podStartSLOduration=1.556199344 podStartE2EDuration="1.556199344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:01.546101773 +0000 UTC m=+821.737939395" watchObservedRunningTime="2026-09-09 20:54:01.556199344 +0000 UTC m=+821.748036943"
Sep 09 20:54:02.380694 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:02.380662    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:03.181286 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:03.181252    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:03.299535 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:03.299505    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:03.509286 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:03.509255    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:03.680574 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:03.680538    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:03.870769 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:03.870672    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:04.079589 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:04.079556    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:04.250346 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:04.250297    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:05.399365 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:05.399299    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:05.507525 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:05.507499    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 20:54:05.585327 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:05.585264    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:06.158363 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.158324    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:06.456743 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.456663    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"]
Sep 09 20:54:06.460290 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.460274    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:06.463454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.463421    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-864e4e\""
Sep 09 20:54:06.463454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.463445    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-864e4e-dockercfg-gjx5p\""
Sep 09 20:54:06.490331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.489656    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"]
Sep 09 20:54:06.569921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.569890    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0c0e2-4028-4289-97c0-31f5eaded0c9-plugins-config-volume\") pod \"payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk\" (UID: \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\") " pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:06.569921 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.569927    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxcbs\" (UniqueName: \"kubernetes.io/projected/70e0c0e2-4028-4289-97c0-31f5eaded0c9-kube-api-access-pxcbs\") pod \"payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk\" (UID: \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\") " pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:06.670600 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.670562    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0c0e2-4028-4289-97c0-31f5eaded0c9-plugins-config-volume\") pod \"payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk\" (UID: \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\") " pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:06.670600 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.670603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxcbs\" (UniqueName: \"kubernetes.io/projected/70e0c0e2-4028-4289-97c0-31f5eaded0c9-kube-api-access-pxcbs\") pod \"payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk\" (UID: \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\") " pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:06.671244 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.671222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0c0e2-4028-4289-97c0-31f5eaded0c9-plugins-config-volume\") pod \"payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk\" (UID: \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\") " pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:06.684362 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.684334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxcbs\" (UniqueName: \"kubernetes.io/projected/70e0c0e2-4028-4289-97c0-31f5eaded0c9-kube-api-access-pxcbs\") pod \"payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk\" (UID: \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\") " pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:06.758830 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.758795    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:06.770516 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:06.770484    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:07.058684 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:07.058621    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"]
Sep 09 20:54:07.061020 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:54:07.060984    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e0c0e2_4028_4289_97c0_31f5eaded0c9.slice/crio-9709b70fa19b3134d534a7823bda51e3f724f4c06a4d33e9ea5a0099480a7cb2 WatchSource:0}: Error finding container 9709b70fa19b3134d534a7823bda51e3f724f4c06a4d33e9ea5a0099480a7cb2: Status 404 returned error can't find the container with id 9709b70fa19b3134d534a7823bda51e3f724f4c06a4d33e9ea5a0099480a7cb2
Sep 09 20:54:07.327265 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:07.327171    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:07.484464 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:07.484430    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:07.555898 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:07.555864    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk" event={"ID":"70e0c0e2-4028-4289-97c0-31f5eaded0c9","Type":"ContainerStarted","Data":"d86f1b18ff45ff608fbeda3c5f4dd8a5359ab2237c5dfc8631ba8a023329c805"}
Sep 09 20:54:07.556058 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:07.555921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk" event={"ID":"70e0c0e2-4028-4289-97c0-31f5eaded0c9","Type":"ContainerStarted","Data":"9709b70fa19b3134d534a7823bda51e3f724f4c06a4d33e9ea5a0099480a7cb2"}
Sep 09 20:54:07.556058 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:07.555962    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:07.580988 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:07.580894    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk" podStartSLOduration=1.580874896 podStartE2EDuration="1.580874896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:07.579551806 +0000 UTC m=+827.771389433" watchObservedRunningTime="2026-09-09 20:54:07.580874896 +0000 UTC m=+827.772712496"
Sep 09 20:54:08.490742 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:08.490660    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 20:54:08.491188 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:08.490842    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 20:54:08.545330 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:08.545271    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:08.642334 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:08.638171    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:09.148945 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.148901    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:09.247848 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.247807    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:09.301889 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.301858    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"]
Sep 09 20:54:09.305336 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.305298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:09.308427 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.308408    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-04cfc947-dockercfg-8rf78\""
Sep 09 20:54:09.308550 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.308436    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-04cfc947\""
Sep 09 20:54:09.326566 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.326537    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"]
Sep 09 20:54:09.395775 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.395748    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9ee34-599e-4c57-9acc-e1653c6e5d35-plugins-config-volume\") pod \"payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk\" (UID: \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\") " pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:09.395933 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.395801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w9t4\" (UniqueName: \"kubernetes.io/projected/33b9ee34-599e-4c57-9acc-e1653c6e5d35-kube-api-access-6w9t4\") pod \"payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk\" (UID: \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\") " pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:09.497230 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.497200    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w9t4\" (UniqueName: \"kubernetes.io/projected/33b9ee34-599e-4c57-9acc-e1653c6e5d35-kube-api-access-6w9t4\") pod \"payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk\" (UID: \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\") " pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:09.497611 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.497289    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9ee34-599e-4c57-9acc-e1653c6e5d35-plugins-config-volume\") pod \"payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk\" (UID: \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\") " pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:09.497960 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.497941    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9ee34-599e-4c57-9acc-e1653c6e5d35-plugins-config-volume\") pod \"payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk\" (UID: \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\") " pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:09.509433 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.509407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w9t4\" (UniqueName: \"kubernetes.io/projected/33b9ee34-599e-4c57-9acc-e1653c6e5d35-kube-api-access-6w9t4\") pod \"payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk\" (UID: \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\") " pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:09.615491 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.615461    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:09.748060 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.747847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:09.911414 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:09.911385    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"]
Sep 09 20:54:09.913849 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:54:09.913823    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b9ee34_599e_4c57_9acc_e1653c6e5d35.slice/crio-607a9cf71fef79dac9332d277170e34eff9ff1d6ece5d14488cf90703f4d2737 WatchSource:0}: Error finding container 607a9cf71fef79dac9332d277170e34eff9ff1d6ece5d14488cf90703f4d2737: Status 404 returned error can't find the container with id 607a9cf71fef79dac9332d277170e34eff9ff1d6ece5d14488cf90703f4d2737
Sep 09 20:54:10.573393 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:10.573350    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk" event={"ID":"33b9ee34-599e-4c57-9acc-e1653c6e5d35","Type":"ContainerStarted","Data":"a33279124b6475e1721c04be7d553600b3e89f8b1feffa02ce8204a417675eea"}
Sep 09 20:54:10.573766 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:10.573546    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:10.573766 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:10.573581    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk" event={"ID":"33b9ee34-599e-4c57-9acc-e1653c6e5d35","Type":"ContainerStarted","Data":"607a9cf71fef79dac9332d277170e34eff9ff1d6ece5d14488cf90703f4d2737"}
Sep 09 20:54:10.626234 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:10.626182    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk" podStartSLOduration=1.6261672150000002 podStartE2EDuration="1.626167215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:10.624346466 +0000 UTC m=+830.816184066" watchObservedRunningTime="2026-09-09 20:54:10.626167215 +0000 UTC m=+830.818004847"
Sep 09 20:54:11.093107 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:11.093059    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:11.366497 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:11.366393    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:11.459617 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:11.459572    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:11.511237 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:11.511210    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 20:54:11.639858 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:11.639776    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:12.516750 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:12.516722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 20:54:13.468126 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:13.468060    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:54:13.591567 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:13.591498    2566 generic.go:356] "Generic (PLEG): container finished" podID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerID="9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6" exitCode=0
Sep 09 20:54:13.591567 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:13.591534    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" event={"ID":"ffa8e0d1-e6d1-4678-b4c6-c99a96981487","Type":"ContainerDied","Data":"9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6"}
Sep 09 20:54:13.591870 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:13.591856    2566 scope.go:117] "RemoveContainer" containerID="9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6"
Sep 09 20:54:13.592905 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:13.592880    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:54:13.592983 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:13.592944    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:54:13.592983 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:13.592974    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 20:54:14.259028 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:14.258994    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:14.598858 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:14.598784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" event={"ID":"ffa8e0d1-e6d1-4678-b4c6-c99a96981487","Type":"ContainerStarted","Data":"e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190"}
Sep 09 20:54:14.849413 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:14.849284    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:54:14.849617 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:14.849567    2566 patch_prober.go:28] interesting pod/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:54:14.849689 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:14.849645    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 09 20:54:15.848479 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:15.848446    2566 patch_prober.go:28] interesting pod/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:54:15.848861 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:15.848502    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 09 20:54:15.922238 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:15.922194    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:16.848465 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:16.848425    2566 patch_prober.go:28] interesting pod/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:54:16.848630 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:16.848515    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 09 20:54:17.333721 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:17.333678    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:17.849066 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:17.849022    2566 patch_prober.go:28] interesting pod/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:54:17.849587 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:17.849103    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 09 20:54:18.564411 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:18.564374    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:54:18.848721 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:18.848621    2566 patch_prober.go:28] interesting pod/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused" start-of-body=
Sep 09 20:54:18.848721 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:18.848689    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.36:15021/healthz/ready\": dial tcp 10.132.0.36:15021: connect: connection refused"
Sep 09 20:54:19.671917 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:19.671881    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:20.009447 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:20.009415    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:54:20.009795 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:20.009752    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:54:20.010960 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:20.010940    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 20:54:21.141821 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:21.141783    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:21.581145 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:21.581113    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:22.041920 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:22.041876    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:22.961362 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:22.961326    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.619458 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:23.619417    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.795958 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:23.795922    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:23.920668 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:23.920589    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:24.017687 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:24.017653    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:24.532984 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:24.532951    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:24.744608 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:24.744573    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:24.935581 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:24.935499    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:27.347397 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:27.347366    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:27.993330 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:27.993284    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:28.107349 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:28.107292    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:28.593720 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:28.593685    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:28.762634 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:28.762600    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:28.862185 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:28.862107    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:29.446576 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:29.446538    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:29.560994 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:29.560961    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:29.682039 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:29.682000    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:30.350263 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.350232    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:30.461795 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.461767    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:30.527505 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.527472    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"]
Sep 09 20:54:30.530454 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.530439    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:30.532947 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.532915    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-423e6a\""
Sep 09 20:54:30.532947 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.532937    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-423e6a-dockercfg-7vzxs\""
Sep 09 20:54:30.545329 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.545157    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"]
Sep 09 20:54:30.693649 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.693558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223badda-803d-469f-885b-2091d943bbf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv\" (UID: \"223badda-803d-469f-885b-2091d943bbf9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:30.693649 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.693607    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjbzq\" (UniqueName: \"kubernetes.io/projected/223badda-803d-469f-885b-2091d943bbf9-kube-api-access-bjbzq\") pod \"payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv\" (UID: \"223badda-803d-469f-885b-2091d943bbf9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:30.794382 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.794349    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223badda-803d-469f-885b-2091d943bbf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv\" (UID: \"223badda-803d-469f-885b-2091d943bbf9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:30.794578 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.794390    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjbzq\" (UniqueName: \"kubernetes.io/projected/223badda-803d-469f-885b-2091d943bbf9-kube-api-access-bjbzq\") pod \"payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv\" (UID: \"223badda-803d-469f-885b-2091d943bbf9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:30.795050 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.795028    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223badda-803d-469f-885b-2091d943bbf9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv\" (UID: \"223badda-803d-469f-885b-2091d943bbf9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:30.802857 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.802835    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjbzq\" (UniqueName: \"kubernetes.io/projected/223badda-803d-469f-885b-2091d943bbf9-kube-api-access-bjbzq\") pod \"payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv\" (UID: \"223badda-803d-469f-885b-2091d943bbf9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:30.840843 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:30.840820    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:31.112930 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:31.112909    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"]
Sep 09 20:54:31.114895 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:54:31.114866    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod223badda_803d_469f_885b_2091d943bbf9.slice/crio-d2bce6906761c2124d629ee92255448194eed60bfcdf0ed635cfb3a2a85375d7 WatchSource:0}: Error finding container d2bce6906761c2124d629ee92255448194eed60bfcdf0ed635cfb3a2a85375d7: Status 404 returned error can't find the container with id d2bce6906761c2124d629ee92255448194eed60bfcdf0ed635cfb3a2a85375d7
Sep 09 20:54:31.480335 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:31.480274    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:54:31.480629 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:31.480591    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:31.689017 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:31.688982    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv" event={"ID":"223badda-803d-469f-885b-2091d943bbf9","Type":"ContainerStarted","Data":"4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4"}
Sep 09 20:54:31.689217 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:31.689026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv" event={"ID":"223badda-803d-469f-885b-2091d943bbf9","Type":"ContainerStarted","Data":"d2bce6906761c2124d629ee92255448194eed60bfcdf0ed635cfb3a2a85375d7"}
Sep 09 20:54:31.689217 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:31.689122    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:31.712960 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:31.712918    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv" podStartSLOduration=1.7129062830000001 podStartE2EDuration="1.712906283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:31.712677117 +0000 UTC m=+851.904514715" watchObservedRunningTime="2026-09-09 20:54:31.712906283 +0000 UTC m=+851.904743883"
Sep 09 20:54:34.849354 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:34.849298    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:34.952628 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:34.952592    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:35.047356 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:35.047287    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:35.229437 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:35.229412    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:37.899641 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:37.899611    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"]
Sep 09 20:54:37.902750 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:37.902733    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:37.905644 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:37.905621    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-47e41c\"/\"kube-root-ca.crt\""
Sep 09 20:54:37.905764 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:37.905630    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-47e41c\"/\"openshift-service-ca.crt\""
Sep 09 20:54:37.905930 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:37.905892    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-47e41c\"/\"default-dockercfg-cbzth\""
Sep 09 20:54:37.906474 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:37.906456    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-47e41c\"/\"simulator-w4-47e41c-kserve-self-signed-certs\""
Sep 09 20:54:37.919652 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:37.919630    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"]
Sep 09 20:54:37.929809 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:37.929775    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:38.059273 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.059243    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-home\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.059481 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.059281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-model-cache\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.059481 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.059387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tmp-dir\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.059481 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.059417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-dshm\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.059481 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.059436    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xrns\" (UniqueName: \"kubernetes.io/projected/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-kube-api-access-8xrns\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.059637 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.059496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tls-certs\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.160494 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.160412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-home\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.160494 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.160452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-model-cache\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.160705 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.160497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tmp-dir\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.160705 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.160519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-dshm\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.160705 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.160535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xrns\" (UniqueName: \"kubernetes.io/projected/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-kube-api-access-8xrns\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.160705 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.160579    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tls-certs\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.160896 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.160870    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-home\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.160958 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.160893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tmp-dir\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.160958 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.160943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-model-cache\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.162873 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.162848    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-dshm\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.163121 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.163104    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tls-certs\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.169573 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.169551    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xrns\" (UniqueName: \"kubernetes.io/projected/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-kube-api-access-8xrns\") pod \"simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") " pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.212451 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.212432    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:38.486521 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.486496    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"]
Sep 09 20:54:38.723965 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.723929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts" event={"ID":"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d","Type":"ContainerStarted","Data":"533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a"}
Sep 09 20:54:38.724141 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.723974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts" event={"ID":"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d","Type":"ContainerStarted","Data":"e86272c4be2e06793538a33e1829c9ab74d491d2789961df5fbff26c51e5d76d"}
Sep 09 20:54:38.762186 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:38.762095    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts" podStartSLOduration=1.7620807109999999 podStartE2EDuration="1.762080711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:38.758589826 +0000 UTC m=+858.950427427" watchObservedRunningTime="2026-09-09 20:54:38.762080711 +0000 UTC m=+858.953918310"
Sep 09 20:54:41.558847 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:41.558800    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:41.641428 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:41.641393    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:42.696365 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:42.696336    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 20:54:46.067423 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:46.067384    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:54:48.213193 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:48.213157    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:48.213679 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:48.213213    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:48.225336 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:48.225294    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:48.790202 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:48.790175    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 20:54:49.486877 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.486847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:49.578116 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.578079    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"]
Sep 09 20:54:49.578331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.578271    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk" podUID="33b9ee34-599e-4c57-9acc-e1653c6e5d35" containerName="payload-processing" containerID="cri-o://a33279124b6475e1721c04be7d553600b3e89f8b1feffa02ce8204a417675eea" gracePeriod=30
Sep 09 20:54:49.655434 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.655402    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:49.786612 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.786579    2566 generic.go:356] "Generic (PLEG): container finished" podID="33b9ee34-599e-4c57-9acc-e1653c6e5d35" containerID="a33279124b6475e1721c04be7d553600b3e89f8b1feffa02ce8204a417675eea" exitCode=0
Sep 09 20:54:49.786781 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.786652    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk" event={"ID":"33b9ee34-599e-4c57-9acc-e1653c6e5d35","Type":"ContainerDied","Data":"a33279124b6475e1721c04be7d553600b3e89f8b1feffa02ce8204a417675eea"}
Sep 09 20:54:49.805734 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.805699    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:49.839836 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.839814    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:49.859867 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.859842    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9ee34-599e-4c57-9acc-e1653c6e5d35-plugins-config-volume\") pod \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\" (UID: \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\") "
Sep 09 20:54:49.860030 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.859977    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6w9t4\" (UniqueName: \"kubernetes.io/projected/33b9ee34-599e-4c57-9acc-e1653c6e5d35-kube-api-access-6w9t4\") pod \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\" (UID: \"33b9ee34-599e-4c57-9acc-e1653c6e5d35\") "
Sep 09 20:54:49.860280 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.860243    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b9ee34-599e-4c57-9acc-e1653c6e5d35-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33b9ee34-599e-4c57-9acc-e1653c6e5d35" (UID: "33b9ee34-599e-4c57-9acc-e1653c6e5d35"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:54:49.862205 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.862182    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b9ee34-599e-4c57-9acc-e1653c6e5d35-kube-api-access-6w9t4" (OuterVolumeSpecName: "kube-api-access-6w9t4") pod "33b9ee34-599e-4c57-9acc-e1653c6e5d35" (UID: "33b9ee34-599e-4c57-9acc-e1653c6e5d35"). InnerVolumeSpecName "kube-api-access-6w9t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:54:49.913532 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.913496    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:49.961250 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.961221    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w9t4\" (UniqueName: \"kubernetes.io/projected/33b9ee34-599e-4c57-9acc-e1653c6e5d35-kube-api-access-6w9t4\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:54:49.961250 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:49.961243    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9ee34-599e-4c57-9acc-e1653c6e5d35-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:54:50.005125 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.005040    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:50.111505 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.111470    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:50.611892 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.611815    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:50.717325 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.717285    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:50.792433 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.792405    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"
Sep 09 20:54:50.792433 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.792413    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk" event={"ID":"33b9ee34-599e-4c57-9acc-e1653c6e5d35","Type":"ContainerDied","Data":"607a9cf71fef79dac9332d277170e34eff9ff1d6ece5d14488cf90703f4d2737"}
Sep 09 20:54:50.792663 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.792506    2566 scope.go:117] "RemoveContainer" containerID="a33279124b6475e1721c04be7d553600b3e89f8b1feffa02ce8204a417675eea"
Sep 09 20:54:50.824258 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.824235    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"]
Sep 09 20:54:50.832252 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.832230    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-04cfc947-555754c4cf-zq9bk"]
Sep 09 20:54:50.864571 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.864501    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:50.995086 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:50.995055    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:51.297757 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:51.297727    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:51.670644 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:51.670549    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:52.285877 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:52.285838    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b9ee34-599e-4c57-9acc-e1653c6e5d35" path="/var/lib/kubelet/pods/33b9ee34-599e-4c57-9acc-e1653c6e5d35/volumes"
Sep 09 20:54:52.955814 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:52.955773    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:53.612567 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:53.612537    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:53.736207 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:53.736171    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:53.898629 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:53.898536    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:54.111628 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:54.111600    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:54.488137 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:54.488099    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:54.583651 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:54.583615    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:56.666938 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.666904    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"]
Sep 09 20:54:56.667284 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.667271    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b9ee34-599e-4c57-9acc-e1653c6e5d35" containerName="payload-processing"
Sep 09 20:54:56.667369 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.667285    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b9ee34-599e-4c57-9acc-e1653c6e5d35" containerName="payload-processing"
Sep 09 20:54:56.667413 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.667374    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b9ee34-599e-4c57-9acc-e1653c6e5d35" containerName="payload-processing"
Sep 09 20:54:56.671889 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.671869    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.679270 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.679242    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d36b91\"/\"openshift-service-ca.crt\""
Sep 09 20:54:56.679425 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.679270    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d36b91\"/\"default-dockercfg-cw4tw\""
Sep 09 20:54:56.679425 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.679409    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d36b91\"/\"simulator-w0-d36b91-kserve-self-signed-certs\""
Sep 09 20:54:56.679547 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.679530    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-d36b91\"/\"kube-root-ca.crt\""
Sep 09 20:54:56.682088 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.682068    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"]
Sep 09 20:54:56.716476 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.716457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tls-certs\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.716576 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.716485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6rxq\" (UniqueName: \"kubernetes.io/projected/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-kube-api-access-x6rxq\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.716576 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.716503    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-dshm\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.716576 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.716518    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tmp-dir\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.716688 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.716587    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-home\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.716688 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.716611    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-model-cache\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.817748 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.817720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tls-certs\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.817875 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.817755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6rxq\" (UniqueName: \"kubernetes.io/projected/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-kube-api-access-x6rxq\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.817875 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.817775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-dshm\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.817875 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.817793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tmp-dir\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.817875 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.817827    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-home\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.817875 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.817844    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-model-cache\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.818269 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.818246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-model-cache\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.818390 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.818299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tmp-dir\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.818390 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.818352    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-home\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.820028 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.820004    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-dshm\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.820261 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.820244    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tls-certs\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.825240 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.825221    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6rxq\" (UniqueName: \"kubernetes.io/projected/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-kube-api-access-x6rxq\") pod \"simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") " pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:56.988057 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:56.988034    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:54:57.076360 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:57.076294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:54:57.261066 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:57.261038    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"]
Sep 09 20:54:57.263648 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:54:57.263615    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3cd2bcc_7c04_4981_82f4_f21649ec76f3.slice/crio-1251ea9dc30bcd7b0c485d4a6e1d9f833cdb0c3e0fb261521862568958514451 WatchSource:0}: Error finding container 1251ea9dc30bcd7b0c485d4a6e1d9f833cdb0c3e0fb261521862568958514451: Status 404 returned error can't find the container with id 1251ea9dc30bcd7b0c485d4a6e1d9f833cdb0c3e0fb261521862568958514451
Sep 09 20:54:57.829831 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:57.829792    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l" event={"ID":"b3cd2bcc-7c04-4981-82f4-f21649ec76f3","Type":"ContainerStarted","Data":"c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab"}
Sep 09 20:54:57.830169 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:57.829842    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l" event={"ID":"b3cd2bcc-7c04-4981-82f4-f21649ec76f3","Type":"ContainerStarted","Data":"1251ea9dc30bcd7b0c485d4a6e1d9f833cdb0c3e0fb261521862568958514451"}
Sep 09 20:54:57.851810 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:54:57.851763    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l" podStartSLOduration=1.851748872 podStartE2EDuration="1.851748872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:54:57.850040772 +0000 UTC m=+878.041878362" watchObservedRunningTime="2026-09-09 20:54:57.851748872 +0000 UTC m=+878.043586471"
Sep 09 20:55:02.661910 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:02.661875    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:55:06.013843 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:06.013806    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:06.117889 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:06.117856    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:06.988958 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:06.988925    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:55:06.989285 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:06.989261    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:55:07.001966 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:07.001942    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:55:07.892916 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:07.892890    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 20:55:12.554627 ip-10-0-137-187 kubenswrapper[2566]: E0909 20:55:12.554603    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/systemd-tmpfiles-clean.service\": RecentStats: unable to find data in memory cache]"
Sep 09 20:55:24.570860 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:24.570820    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:24.672680 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:24.672635    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:24.771349 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:24.771287    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:24.835764 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:24.835664    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:24.947741 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:24.947696    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:25.429341 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:25.429300    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:25.731050 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:25.731020    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:25.895681 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:25.895649    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:28.677494 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:28.677459    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:28.766899 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:28.766859    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:29.843579 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:29.843539    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:29.947575 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:29.947537    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:31.036053 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:31.036021    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:31.136382 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:31.136340    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:32.155239 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:32.155208    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:55:42.779731 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.779696    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"]
Sep 09 20:55:42.784789 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.784768    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.788006 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.787970    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-864e4e\"/\"autores-model-864e4e-kserve-self-signed-certs\""
Sep 09 20:55:42.788006 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.787990    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-864e4e\"/\"openshift-service-ca.crt\""
Sep 09 20:55:42.788188 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.788041    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-864e4e\"/\"default-dockercfg-qqn85\""
Sep 09 20:55:42.788188 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.788042    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-864e4e\"/\"kube-root-ca.crt\""
Sep 09 20:55:42.794100 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.794080    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"]
Sep 09 20:55:42.813046 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.813012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jxlk\" (UniqueName: \"kubernetes.io/projected/03a249c8-7dd5-4154-aea1-079a82acf012-kube-api-access-6jxlk\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.813172 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.813121    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a249c8-7dd5-4154-aea1-079a82acf012-tls-certs\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.813172 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.813159    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-tmp-dir\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.813303 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.813216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-dshm\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.813303 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.813244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-home\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.813303 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.813279    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-model-cache\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.876388 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.876358    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:42.914407 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.914382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jxlk\" (UniqueName: \"kubernetes.io/projected/03a249c8-7dd5-4154-aea1-079a82acf012-kube-api-access-6jxlk\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.914532 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.914434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a249c8-7dd5-4154-aea1-079a82acf012-tls-certs\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.914592 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.914546    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-tmp-dir\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.914648 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.914596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-dshm\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.914648 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.914613    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-home\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.914743 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.914707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-model-cache\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.914937 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.914912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-tmp-dir\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.915056 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.914970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-home\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.915117 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.915040    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-model-cache\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.916658 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.916641    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-dshm\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.916769 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.916753    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a249c8-7dd5-4154-aea1-079a82acf012-tls-certs\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:42.922359 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:42.922337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jxlk\" (UniqueName: \"kubernetes.io/projected/03a249c8-7dd5-4154-aea1-079a82acf012-kube-api-access-6jxlk\") pod \"autores-model-864e4e-kserve-5d664c5c8c-7jzl6\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") " pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:43.094241 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:43.094161    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:43.372860 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:43.372831    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"]
Sep 09 20:55:43.374224 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:55:43.374198    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod03a249c8_7dd5_4154_aea1_079a82acf012.slice/crio-542435df6c8b9f5536c6dd0878ce2fa4e8b22e9c0e399b292196b4c439cd064a WatchSource:0}: Error finding container 542435df6c8b9f5536c6dd0878ce2fa4e8b22e9c0e399b292196b4c439cd064a: Status 404 returned error can't find the container with id 542435df6c8b9f5536c6dd0878ce2fa4e8b22e9c0e399b292196b4c439cd064a
Sep 09 20:55:44.056526 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:44.056495    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6" event={"ID":"03a249c8-7dd5-4154-aea1-079a82acf012","Type":"ContainerStarted","Data":"ecb1178cb89ef7f496d2ce4fc8e667dd5dab5034ff5ae025ba80620d92e65382"}
Sep 09 20:55:44.056922 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:44.056536    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6" event={"ID":"03a249c8-7dd5-4154-aea1-079a82acf012","Type":"ContainerStarted","Data":"542435df6c8b9f5536c6dd0878ce2fa4e8b22e9c0e399b292196b4c439cd064a"}
Sep 09 20:55:44.074834 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:44.074784    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6" podStartSLOduration=2.074771106 podStartE2EDuration="2.074771106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:55:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:55:44.072067499 +0000 UTC m=+924.263905099" watchObservedRunningTime="2026-09-09 20:55:44.074771106 +0000 UTC m=+924.266608725"
Sep 09 20:55:49.267834 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:49.267799    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:55:53.094666 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:53.094632    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:53.094666 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:53.094674    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:53.107828 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:53.107805    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:53.118905 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:53.118880    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:55:53.937771 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:53.937732    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:55:56.240360 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:55:56.240328    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:00.385122 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:00.385087    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:00.482966 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:00.482928    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:00.582750 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:00.582717    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:00.684231 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:00.684143    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:01.726350 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:01.726293    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:02.147649 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:02.147611    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:02.547156 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:02.547123    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:03.053774 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:03.053739    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:04.972301 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:04.972269    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:05.069345 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:05.069294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:06.127265 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:06.127222    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:06.233745 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:06.233707    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:06.753021 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:06.752986    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:06.868183 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:06.868149    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:06.915892 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:06.915858    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:56:15.018419 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:15.018388    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:17.515109 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:17.515067    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:24.464822 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:24.464787    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:56:28.097705 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.097668    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:28.121267 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.121240    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"]
Sep 09 20:56:28.121514 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.121488    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk" podUID="70e0c0e2-4028-4289-97c0-31f5eaded0c9" containerName="payload-processing" containerID="cri-o://d86f1b18ff45ff608fbeda3c5f4dd8a5359ab2237c5dfc8631ba8a023329c805" gracePeriod=30
Sep 09 20:56:28.281826 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.281791    2566 generic.go:356] "Generic (PLEG): container finished" podID="70e0c0e2-4028-4289-97c0-31f5eaded0c9" containerID="d86f1b18ff45ff608fbeda3c5f4dd8a5359ab2237c5dfc8631ba8a023329c805" exitCode=0
Sep 09 20:56:28.286096 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.286067    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk" event={"ID":"70e0c0e2-4028-4289-97c0-31f5eaded0c9","Type":"ContainerDied","Data":"d86f1b18ff45ff608fbeda3c5f4dd8a5359ab2237c5dfc8631ba8a023329c805"}
Sep 09 20:56:28.378748 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.378727    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:56:28.460456 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.460423    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:28.505663 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.505638    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pxcbs\" (UniqueName: \"kubernetes.io/projected/70e0c0e2-4028-4289-97c0-31f5eaded0c9-kube-api-access-pxcbs\") pod \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\" (UID: \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\") "
Sep 09 20:56:28.505814 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.505687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0c0e2-4028-4289-97c0-31f5eaded0c9-plugins-config-volume\") pod \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\" (UID: \"70e0c0e2-4028-4289-97c0-31f5eaded0c9\") "
Sep 09 20:56:28.506066 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.506042    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e0c0e2-4028-4289-97c0-31f5eaded0c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70e0c0e2-4028-4289-97c0-31f5eaded0c9" (UID: "70e0c0e2-4028-4289-97c0-31f5eaded0c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 20:56:28.507627 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.507608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e0c0e2-4028-4289-97c0-31f5eaded0c9-kube-api-access-pxcbs" (OuterVolumeSpecName: "kube-api-access-pxcbs") pod "70e0c0e2-4028-4289-97c0-31f5eaded0c9" (UID: "70e0c0e2-4028-4289-97c0-31f5eaded0c9"). InnerVolumeSpecName "kube-api-access-pxcbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:28.595770 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.595740    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:28.607204 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.607182    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxcbs\" (UniqueName: \"kubernetes.io/projected/70e0c0e2-4028-4289-97c0-31f5eaded0c9-kube-api-access-pxcbs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:56:28.607204 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.607202    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e0c0e2-4028-4289-97c0-31f5eaded0c9-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:56:28.745494 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.745458    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:28.980611 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:28.980579    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:29.288005 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:29.287981    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"
Sep 09 20:56:29.288408 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:29.288026    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk" event={"ID":"70e0c0e2-4028-4289-97c0-31f5eaded0c9","Type":"ContainerDied","Data":"9709b70fa19b3134d534a7823bda51e3f724f4c06a4d33e9ea5a0099480a7cb2"}
Sep 09 20:56:29.288408 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:29.288078    2566 scope.go:117] "RemoveContainer" containerID="d86f1b18ff45ff608fbeda3c5f4dd8a5359ab2237c5dfc8631ba8a023329c805"
Sep 09 20:56:29.310941 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:29.310917    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"]
Sep 09 20:56:29.314352 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:29.314330    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-864e4e-6c8c68d685-bj2rk"]
Sep 09 20:56:30.173320 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:30.173273    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"]
Sep 09 20:56:30.196457 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:30.196429    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:30.286188 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:30.286156    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70e0c0e2-4028-4289-97c0-31f5eaded0c9" path="/var/lib/kubelet/pods/70e0c0e2-4028-4289-97c0-31f5eaded0c9/volumes"
Sep 09 20:56:33.805354 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:33.805298    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:45.193639 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.193534    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6" podUID="03a249c8-7dd5-4154-aea1-079a82acf012" containerName="main" containerID="cri-o://ecb1178cb89ef7f496d2ce4fc8e667dd5dab5034ff5ae025ba80620d92e65382" gracePeriod=45
Sep 09 20:56:45.264556 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.264526    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:56:45.382688 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.382659    2566 generic.go:356] "Generic (PLEG): container finished" podID="03a249c8-7dd5-4154-aea1-079a82acf012" containerID="ecb1178cb89ef7f496d2ce4fc8e667dd5dab5034ff5ae025ba80620d92e65382" exitCode=0
Sep 09 20:56:45.382824 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.382699    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6" event={"ID":"03a249c8-7dd5-4154-aea1-079a82acf012","Type":"ContainerDied","Data":"ecb1178cb89ef7f496d2ce4fc8e667dd5dab5034ff5ae025ba80620d92e65382"}
Sep 09 20:56:45.440385 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.440364    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:56:45.543599 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.543564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-home\") pod \"03a249c8-7dd5-4154-aea1-079a82acf012\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") "
Sep 09 20:56:45.543759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.543623    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-tmp-dir\") pod \"03a249c8-7dd5-4154-aea1-079a82acf012\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") "
Sep 09 20:56:45.543759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.543665    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-dshm\") pod \"03a249c8-7dd5-4154-aea1-079a82acf012\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") "
Sep 09 20:56:45.543759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.543687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jxlk\" (UniqueName: \"kubernetes.io/projected/03a249c8-7dd5-4154-aea1-079a82acf012-kube-api-access-6jxlk\") pod \"03a249c8-7dd5-4154-aea1-079a82acf012\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") "
Sep 09 20:56:45.543759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.543705    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a249c8-7dd5-4154-aea1-079a82acf012-tls-certs\") pod \"03a249c8-7dd5-4154-aea1-079a82acf012\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") "
Sep 09 20:56:45.543759 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.543734    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-model-cache\") pod \"03a249c8-7dd5-4154-aea1-079a82acf012\" (UID: \"03a249c8-7dd5-4154-aea1-079a82acf012\") "
Sep 09 20:56:45.544108 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.543808    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-home" (OuterVolumeSpecName: "home") pod "03a249c8-7dd5-4154-aea1-079a82acf012" (UID: "03a249c8-7dd5-4154-aea1-079a82acf012"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:45.544108 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.543843    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "03a249c8-7dd5-4154-aea1-079a82acf012" (UID: "03a249c8-7dd5-4154-aea1-079a82acf012"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:45.544108 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.544064    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:56:45.544108 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.544078    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:56:45.544108 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.544077    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-model-cache" (OuterVolumeSpecName: "model-cache") pod "03a249c8-7dd5-4154-aea1-079a82acf012" (UID: "03a249c8-7dd5-4154-aea1-079a82acf012"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:45.545822 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.545800    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-dshm" (OuterVolumeSpecName: "dshm") pod "03a249c8-7dd5-4154-aea1-079a82acf012" (UID: "03a249c8-7dd5-4154-aea1-079a82acf012"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 20:56:45.545924 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.545819    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a249c8-7dd5-4154-aea1-079a82acf012-kube-api-access-6jxlk" (OuterVolumeSpecName: "kube-api-access-6jxlk") pod "03a249c8-7dd5-4154-aea1-079a82acf012" (UID: "03a249c8-7dd5-4154-aea1-079a82acf012"). InnerVolumeSpecName "kube-api-access-6jxlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 20:56:45.545924 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.545896    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03a249c8-7dd5-4154-aea1-079a82acf012-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "03a249c8-7dd5-4154-aea1-079a82acf012" (UID: "03a249c8-7dd5-4154-aea1-079a82acf012"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 20:56:45.645374 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.645339    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:56:45.645374 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.645375    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03a249c8-7dd5-4154-aea1-079a82acf012-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:56:45.645519 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.645390    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jxlk\" (UniqueName: \"kubernetes.io/projected/03a249c8-7dd5-4154-aea1-079a82acf012-kube-api-access-6jxlk\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:56:45.645519 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:45.645404    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03a249c8-7dd5-4154-aea1-079a82acf012-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 20:56:46.389089 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:46.389056    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"
Sep 09 20:56:46.389089 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:46.389073    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6" event={"ID":"03a249c8-7dd5-4154-aea1-079a82acf012","Type":"ContainerDied","Data":"542435df6c8b9f5536c6dd0878ce2fa4e8b22e9c0e399b292196b4c439cd064a"}
Sep 09 20:56:46.389579 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:46.389129    2566 scope.go:117] "RemoveContainer" containerID="ecb1178cb89ef7f496d2ce4fc8e667dd5dab5034ff5ae025ba80620d92e65382"
Sep 09 20:56:46.406467 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:46.406439    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"]
Sep 09 20:56:46.410439 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:46.410418    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-864e4e/autores-model-864e4e-kserve-5d664c5c8c-7jzl6"]
Sep 09 20:56:46.718578 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:46.718497    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:46.817723 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:46.817691    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:46.919838 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:46.919802    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:47.018331 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:47.018286    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:47.148994 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:47.148961    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:47.808004 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:47.807972    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:47.893841 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:47.893808    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:47.974159 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:47.974124    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:48.287216 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:48.287185    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a249c8-7dd5-4154-aea1-079a82acf012" path="/var/lib/kubelet/pods/03a249c8-7dd5-4154-aea1-079a82acf012/volumes"
Sep 09 20:56:49.353492 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:49.353461    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:55.489418 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:55.489378    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:55.603180 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:55.603149    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:55.812535 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:55.812417    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:55.897751 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:55.897719    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:56.205437 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:56.205336    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:57.121398 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:57.121363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:58.703648 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:58.703613    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:59.068358 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:59.068328    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:59.516686 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:59.516656    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:56:59.889499 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:56:59.889406    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:00.208397 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:00.208288    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:02.846771 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:02.846740    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:02.948692 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:02.948661    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:04.214785 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:04.214755    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:04.313922 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:04.313877    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:04.783907 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:04.783868    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:04.883611 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:04.883570    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:05.977717 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:05.977678    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:57:14.089078 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.089049    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"]
Sep 09 20:57:14.089546 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.089529    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e0c0e2-4028-4289-97c0-31f5eaded0c9" containerName="payload-processing"
Sep 09 20:57:14.089597 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.089548    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e0c0e2-4028-4289-97c0-31f5eaded0c9" containerName="payload-processing"
Sep 09 20:57:14.089597 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.089557    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a249c8-7dd5-4154-aea1-079a82acf012" containerName="main"
Sep 09 20:57:14.089597 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.089563    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a249c8-7dd5-4154-aea1-079a82acf012" containerName="main"
Sep 09 20:57:14.089703 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.089635    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e0c0e2-4028-4289-97c0-31f5eaded0c9" containerName="payload-processing"
Sep 09 20:57:14.089703 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.089644    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a249c8-7dd5-4154-aea1-079a82acf012" containerName="main"
Sep 09 20:57:14.092623 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.092608    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.095053 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.095029    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d36b91\"/\"premium-w0-d36b91-kserve-self-signed-certs\""
Sep 09 20:57:14.105616 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.105595    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"]
Sep 09 20:57:14.129302 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.129274    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:14.183602 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.183579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-model-cache\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.183709 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.183629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-dshm\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.183756 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.183708    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-tmp-dir\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.183805 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.183751    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc04ede-7549-48d2-ae86-0e9472d263ca-tls-certs\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.183805 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.183772    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-home\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.183909 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.183851    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5d2h\" (UniqueName: \"kubernetes.io/projected/ddc04ede-7549-48d2-ae86-0e9472d263ca-kube-api-access-t5d2h\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.284710 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.284677    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5d2h\" (UniqueName: \"kubernetes.io/projected/ddc04ede-7549-48d2-ae86-0e9472d263ca-kube-api-access-t5d2h\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.284889 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.284724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-model-cache\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.284889 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.284775    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-dshm\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.284889 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.284839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-tmp-dir\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.284889 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.284878    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc04ede-7549-48d2-ae86-0e9472d263ca-tls-certs\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.285098 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.284903    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-home\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.285223 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.285200    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-model-cache\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.285297 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.285269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-tmp-dir\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.285372 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.285300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-home\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.287204 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.287183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-dshm\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.287303 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.287282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc04ede-7549-48d2-ae86-0e9472d263ca-tls-certs\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.293323 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.293283    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5d2h\" (UniqueName: \"kubernetes.io/projected/ddc04ede-7549-48d2-ae86-0e9472d263ca-kube-api-access-t5d2h\") pod \"premium-w0-d36b91-kserve-6bcf747c-qt6br\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") " pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.402822 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.402747    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:14.674816 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:14.674792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"]
Sep 09 20:57:14.676921 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:57:14.676894    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podddc04ede_7549_48d2_ae86_0e9472d263ca.slice/crio-cf34c2bc013a846ac29277df720cc1c03ea04b7021c54c7d24134fb0d1c97433 WatchSource:0}: Error finding container cf34c2bc013a846ac29277df720cc1c03ea04b7021c54c7d24134fb0d1c97433: Status 404 returned error can't find the container with id cf34c2bc013a846ac29277df720cc1c03ea04b7021c54c7d24134fb0d1c97433
Sep 09 20:57:15.535656 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:15.535618    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br" event={"ID":"ddc04ede-7549-48d2-ae86-0e9472d263ca","Type":"ContainerStarted","Data":"a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892"}
Sep 09 20:57:15.535656 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:15.535661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br" event={"ID":"ddc04ede-7549-48d2-ae86-0e9472d263ca","Type":"ContainerStarted","Data":"cf34c2bc013a846ac29277df720cc1c03ea04b7021c54c7d24134fb0d1c97433"}
Sep 09 20:57:15.553932 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:15.553886    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br" podStartSLOduration=1.5538727620000001 podStartE2EDuration="1.553872762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:15.552653354 +0000 UTC m=+1015.744490953" watchObservedRunningTime="2026-09-09 20:57:15.553872762 +0000 UTC m=+1015.745710361"
Sep 09 20:57:22.145082 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:22.145051    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:24.402845 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:24.402819    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:24.403208 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:24.402861    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:24.415256 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:24.415230    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:24.592644 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:24.592613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 20:57:27.830712 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:27.830679    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:28.329957 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:28.329920    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:28.457524 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:28.457491    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:28.553928 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:28.553895    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:32.391944 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:32.391888    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:32.954713 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:32.954676    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:57:38.184425 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.184388    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"]
Sep 09 20:57:38.189275 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.189256    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.191758 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.191733    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-423e6a\"/\"kube-root-ca.crt\""
Sep 09 20:57:38.192903 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.192884    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-423e6a\"/\"default-dockercfg-hbl7l\""
Sep 09 20:57:38.192971 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.192932    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-423e6a\"/\"test-model-423e6a-kserve-self-signed-certs\""
Sep 09 20:57:38.192971 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.192950    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-423e6a\"/\"openshift-service-ca.crt\""
Sep 09 20:57:38.202160 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.202140    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"]
Sep 09 20:57:38.291210 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.291183    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f857a1dd-621f-4a07-b818-a277c7b9537b-tls-certs\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.291393 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.291215    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7vv6\" (UniqueName: \"kubernetes.io/projected/f857a1dd-621f-4a07-b818-a277c7b9537b-kube-api-access-j7vv6\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.291393 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.291236    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-home\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.291393 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.291254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-model-cache\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.291393 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.291289    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-tmp-dir\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.291544 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.291449    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-dshm\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.376526 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.376496    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:38.392674 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.392642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-dshm\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.392864 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.392688    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f857a1dd-621f-4a07-b818-a277c7b9537b-tls-certs\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.392864 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.392707    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7vv6\" (UniqueName: \"kubernetes.io/projected/f857a1dd-621f-4a07-b818-a277c7b9537b-kube-api-access-j7vv6\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.392864 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.392732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-home\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.392864 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.392754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-model-cache\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.392864 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.392777    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-tmp-dir\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.393206 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.393171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-home\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.393206 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.393190    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-model-cache\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.393398 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.393243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-tmp-dir\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.394946 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.394924    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-dshm\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.395232 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.395213    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f857a1dd-621f-4a07-b818-a277c7b9537b-tls-certs\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.433907 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.433884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7vv6\" (UniqueName: \"kubernetes.io/projected/f857a1dd-621f-4a07-b818-a277c7b9537b-kube-api-access-j7vv6\") pod \"test-model-423e6a-kserve-7dcff7db75-cmvd5\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") " pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.498969 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.498943    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:38.780960 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:57:38.780672    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf857a1dd_621f_4a07_b818_a277c7b9537b.slice/crio-3dec0688b992c3cf3874ff975b0c4783056a70383428f3288f2cf6fa4a2c49b3 WatchSource:0}: Error finding container 3dec0688b992c3cf3874ff975b0c4783056a70383428f3288f2cf6fa4a2c49b3: Status 404 returned error can't find the container with id 3dec0688b992c3cf3874ff975b0c4783056a70383428f3288f2cf6fa4a2c49b3
Sep 09 20:57:38.784502 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:38.784018    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"]
Sep 09 20:57:39.655540 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:39.655507    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5" event={"ID":"f857a1dd-621f-4a07-b818-a277c7b9537b","Type":"ContainerStarted","Data":"e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1"}
Sep 09 20:57:39.655540 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:39.655545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5" event={"ID":"f857a1dd-621f-4a07-b818-a277c7b9537b","Type":"ContainerStarted","Data":"3dec0688b992c3cf3874ff975b0c4783056a70383428f3288f2cf6fa4a2c49b3"}
Sep 09 20:57:39.676295 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:39.676247    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5" podStartSLOduration=1.676234051 podStartE2EDuration="1.676234051s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:57:39.674573791 +0000 UTC m=+1039.866411402" watchObservedRunningTime="2026-09-09 20:57:39.676234051 +0000 UTC m=+1039.868071650"
Sep 09 20:57:44.382979 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:44.382945    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:48.500118 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:48.500084    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:48.500474 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:48.500141    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:48.512299 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:48.512276    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:48.710770 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:48.710741    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 20:57:50.059851 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:50.059818    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:57:50.555031 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:50.554994    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:50.679590 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:50.679561    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:57:50.763264 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:57:50.763227    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:01.410376 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:01.410345    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:01.491242 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:01.491212    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:02.339421 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:02.339389    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:08.166178 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:08.166147    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:08.288359 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:08.288325    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:08.465993 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:08.465916    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:08.519046 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:08.519012    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:58:08.654994 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:08.654960    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:09.229617 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:09.229587    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:09.332288 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:09.332255    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:09.434727 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:09.434694    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:11.088583 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:11.088548    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:17.241629 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:17.241551    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:17.367460 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:17.367422    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:17.926616 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:17.926574    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:19.590503 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.590455    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"]
Sep 09 20:58:19.595247 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.595220    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.598067 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.598038    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-47e41c\"/\"distinct-w4-47e41c-kserve-self-signed-certs\""
Sep 09 20:58:19.602845 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.602823    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"]
Sep 09 20:58:19.627011 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.626973    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:19.739946 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.739908    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-model-cache\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.740172 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.739977    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-tmp-dir\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.740172 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.740038    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sr6b\" (UniqueName: \"kubernetes.io/projected/27c16589-1774-4286-b956-2324aeb5101d-kube-api-access-7sr6b\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.740172 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.740077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-dshm\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.740298 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.740222    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-home\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.740298 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.740281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c16589-1774-4286-b956-2324aeb5101d-tls-certs\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.841728 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.841646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c16589-1774-4286-b956-2324aeb5101d-tls-certs\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.841882 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.841737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-model-cache\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.841882 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.841787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-tmp-dir\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.841882 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.841825    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sr6b\" (UniqueName: \"kubernetes.io/projected/27c16589-1774-4286-b956-2324aeb5101d-kube-api-access-7sr6b\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.842059 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.841938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-dshm\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.842059 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.842045    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-home\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.842479 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.842453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-tmp-dir\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.842633 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.842491    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-home\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.842760 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.842743    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-model-cache\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.844280 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.844256    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c16589-1774-4286-b956-2324aeb5101d-tls-certs\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.844748 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.844728    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-dshm\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.850443 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.850421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sr6b\" (UniqueName: \"kubernetes.io/projected/27c16589-1774-4286-b956-2324aeb5101d-kube-api-access-7sr6b\") pod \"distinct-w4-47e41c-kserve-85588d6967-mmnfm\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") " pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:19.905421 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:19.905398    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:20.180249 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:20.180219    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"]
Sep 09 20:58:20.183002 ip-10-0-137-187 kubenswrapper[2566]: W0909 20:58:20.182974    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod27c16589_1774_4286_b956_2324aeb5101d.slice/crio-56d6f8eb417ca4c144bd036c2e7cff5cca028628de5974a89fff09da5d14f6ed WatchSource:0}: Error finding container 56d6f8eb417ca4c144bd036c2e7cff5cca028628de5974a89fff09da5d14f6ed: Status 404 returned error can't find the container with id 56d6f8eb417ca4c144bd036c2e7cff5cca028628de5974a89fff09da5d14f6ed
Sep 09 20:58:20.872283 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:20.872245    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm" event={"ID":"27c16589-1774-4286-b956-2324aeb5101d","Type":"ContainerStarted","Data":"ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25"}
Sep 09 20:58:20.872645 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:20.872300    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm" event={"ID":"27c16589-1774-4286-b956-2324aeb5101d","Type":"ContainerStarted","Data":"56d6f8eb417ca4c144bd036c2e7cff5cca028628de5974a89fff09da5d14f6ed"}
Sep 09 20:58:20.900246 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:20.900199    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm" podStartSLOduration=1.900181916 podStartE2EDuration="1.900181916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 20:58:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 20:58:20.898448053 +0000 UTC m=+1081.090285651" watchObservedRunningTime="2026-09-09 20:58:20.900181916 +0000 UTC m=+1081.092019515"
Sep 09 20:58:24.144425 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:24.144393    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:24.217639 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:24.217600    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:24.371435 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:24.371406    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:24.687443 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:24.687409    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:25.099689 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:25.099653    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:28.267011 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:28.266972    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:58:29.905977 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:29.905941    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:29.906449 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:29.905990    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:29.918682 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:29.918657    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:29.929987 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:29.929959    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 20:58:31.243830 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:31.243788    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:31.330953 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:31.330918    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:32.540538 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:32.540501    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:32.716626 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:32.716594    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:33.253398 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:33.253357    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:33.352750 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:33.352717    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:34.933763 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:34.933721    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:45.319857 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:45.319807    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:45.418415 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:45.418378    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:45.505998 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:45.505967    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:45.574639 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:45.574558    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:45.891788 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:45.891714    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:51.112506 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:51.112472    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:53.710420 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:53.710379    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:54.319030 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:54.319000    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:56.255980 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:56.255936    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:57.002459 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:57.002424    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:57.115048 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:57.115010    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:57.464152 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:57.464056    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:58:58.777358 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:58:58.777325    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:59:03.416470 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:03.416436    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:03.482864 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:03.482835    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:14.313030 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:14.312995    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:14.391487 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:14.391452    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:14.461587 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:14.461554    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:14.575439 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:14.575353    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:14.934169 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:14.934079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:18.249788 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:18.249754    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:20.495215 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:20.495175    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:21.166901 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:21.166872    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 20:59:21.336651 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:21.336620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:21.433965 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:21.433879    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:21.706690 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:21.706606    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:21.963732 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:21.963646    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:22.064480 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:22.064447    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:22.166060 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:22.166011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:22.264967 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:22.264930    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:27.998910 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:27.998878    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:28.272936 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:28.272847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:28.571439 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:28.571334    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:28.638883 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:28.638835    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:28.753604 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:28.753571    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:28.930108 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:28.930016    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:29.030591 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:29.030553    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:29.101964 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:29.101935    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:29.328838 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:29.328806    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:30.155027 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:30.154999    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:30.251457 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:30.251427    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:30.360043 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:30.360013    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:32.279548 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:32.279508    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:34.960071 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:34.960037    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:35.035775 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:35.035739    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:35.907598 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:35.907544    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:36.154641 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:36.154601    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:36.889942 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:36.889909    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:38.826903 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:38.826862    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:38.925725 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:38.925689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:39.981617 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:39.981579    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:40.079524 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:40.079486    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:40.257482 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:40.257400    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:40.474439 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:40.474406    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:40.549233 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:40.549135    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:52.442258 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:52.442164    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 20:59:57.663681 ip-10-0-137-187 kubenswrapper[2566]: I0909 20:59:57.663644    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:00:00.150722 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.150686    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz"]
Sep 09 21:00:00.154438 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.154412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz"
Sep 09 21:00:00.156897 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.156871    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-glz44\""
Sep 09 21:00:00.168572 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.168385    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz"
Sep 09 21:00:00.168727 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.168703    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz"]
Sep 09 21:00:00.170794 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.170772    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z"]
Sep 09 21:00:00.174365 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.174348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z"
Sep 09 21:00:00.186104 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.186076    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z"]
Sep 09 21:00:00.203157 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.203137    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z"
Sep 09 21:00:00.447672 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.447650    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz"]
Sep 09 21:00:00.449369 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:00:00.449340    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod099ebc12_123b_43d9_8208_f661340dcadd.slice/crio-b2416a1b574a0c7673fee934d30168f328da7583b98a00d0e143c974655746eb WatchSource:0}: Error finding container b2416a1b574a0c7673fee934d30168f328da7583b98a00d0e143c974655746eb: Status 404 returned error can't find the container with id b2416a1b574a0c7673fee934d30168f328da7583b98a00d0e143c974655746eb
Sep 09 21:00:00.451048 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.451027    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:00:00.488322 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:00.488286    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z"]
Sep 09 21:00:00.489504 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:00:00.489484    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9e9addb_0874_4848_bab8_325d88f63369.slice/crio-8704775ac4b0586a1f3ace5fa8913402d0dc082b411880ab19e8aac399ac990e WatchSource:0}: Error finding container 8704775ac4b0586a1f3ace5fa8913402d0dc082b411880ab19e8aac399ac990e: Status 404 returned error can't find the container with id 8704775ac4b0586a1f3ace5fa8913402d0dc082b411880ab19e8aac399ac990e
Sep 09 21:00:01.379034 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:01.378992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz" event={"ID":"099ebc12-123b-43d9-8208-f661340dcadd","Type":"ContainerStarted","Data":"b2416a1b574a0c7673fee934d30168f328da7583b98a00d0e143c974655746eb"}
Sep 09 21:00:01.381055 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:01.381022    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z" event={"ID":"e9e9addb-0874-4848-bab8-325d88f63369","Type":"ContainerStarted","Data":"8704775ac4b0586a1f3ace5fa8913402d0dc082b411880ab19e8aac399ac990e"}
Sep 09 21:00:02.061276 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:02.061180    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:02.220387 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:02.220339    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:03.394056 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:03.394018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z" event={"ID":"e9e9addb-0874-4848-bab8-325d88f63369","Type":"ContainerStarted","Data":"ebe95b11b3c52139a7abacbed4ccce7d08db8dfd232c9b0d806729403fcabd0b"}
Sep 09 21:00:03.395729 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:03.395706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz" event={"ID":"099ebc12-123b-43d9-8208-f661340dcadd","Type":"ContainerStarted","Data":"6b1309e414167b5ab6828e9720fd7c48b445ffb53b62335139c565fefd623f66"}
Sep 09 21:00:03.420633 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:03.420587    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z" podStartSLOduration=1.261954684 podStartE2EDuration="3.420572686s" totalImagesPullingTime="2.158618002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:03.419776887 +0000 UTC m=+1183.611614493" watchObservedRunningTime="2026-09-09 21:00:03.420572686 +0000 UTC m=+1183.612410284"
Sep 09 21:00:03.439863 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:03.439823    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz" podStartSLOduration=1.245833031 podStartE2EDuration="3.439809157s" totalImagesPullingTime="2.193976126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:03.438871054 +0000 UTC m=+1183.630708653" watchObservedRunningTime="2026-09-09 21:00:03.439809157 +0000 UTC m=+1183.631646756"
Sep 09 21:00:03.837030 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:03.836981    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:04.404057 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:04.404020    2566 generic.go:356] "Generic (PLEG): container finished" podID="e9e9addb-0874-4848-bab8-325d88f63369" containerID="ebe95b11b3c52139a7abacbed4ccce7d08db8dfd232c9b0d806729403fcabd0b" exitCode=0
Sep 09 21:00:04.404503 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:04.404115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z" event={"ID":"e9e9addb-0874-4848-bab8-325d88f63369","Type":"ContainerDied","Data":"ebe95b11b3c52139a7abacbed4ccce7d08db8dfd232c9b0d806729403fcabd0b"}
Sep 09 21:00:04.406258 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:04.406233    2566 generic.go:356] "Generic (PLEG): container finished" podID="099ebc12-123b-43d9-8208-f661340dcadd" containerID="6b1309e414167b5ab6828e9720fd7c48b445ffb53b62335139c565fefd623f66" exitCode=0
Sep 09 21:00:04.406409 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:04.406291    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz" event={"ID":"099ebc12-123b-43d9-8208-f661340dcadd","Type":"ContainerDied","Data":"6b1309e414167b5ab6828e9720fd7c48b445ffb53b62335139c565fefd623f66"}
Sep 09 21:00:04.450490 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:04.450461    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:04.555753 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:04.555719    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:04.822267 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:04.822232    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:05.562894 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:05.562876    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z"
Sep 09 21:00:05.565561 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:05.565531    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz"
Sep 09 21:00:06.419099 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:06.419069    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz"
Sep 09 21:00:06.419282 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:06.419098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29816460-tfxzz" event={"ID":"099ebc12-123b-43d9-8208-f661340dcadd","Type":"ContainerDied","Data":"b2416a1b574a0c7673fee934d30168f328da7583b98a00d0e143c974655746eb"}
Sep 09 21:00:06.419282 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:06.419182    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2416a1b574a0c7673fee934d30168f328da7583b98a00d0e143c974655746eb"
Sep 09 21:00:06.420481 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:06.420459    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z"
Sep 09 21:00:06.420608 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:06.420457    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z" event={"ID":"e9e9addb-0874-4848-bab8-325d88f63369","Type":"ContainerDied","Data":"8704775ac4b0586a1f3ace5fa8913402d0dc082b411880ab19e8aac399ac990e"}
Sep 09 21:00:06.420608 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:06.420576    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8704775ac4b0586a1f3ace5fa8913402d0dc082b411880ab19e8aac399ac990e"
Sep 09 21:00:07.122445 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:07.122401    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:08.128186 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:08.128150    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:08.285549 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:08.285519    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:09.275217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:09.275182    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:09.354117 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:09.354083    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:09.651815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:09.651729    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:09.835303 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:09.835264    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:09.936931 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:09.936847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:10.029739 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.029709    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"]
Sep 09 21:00:10.030101 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.030087    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="099ebc12-123b-43d9-8208-f661340dcadd" containerName="cleanup"
Sep 09 21:00:10.030147 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.030102    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="099ebc12-123b-43d9-8208-f661340dcadd" containerName="cleanup"
Sep 09 21:00:10.030147 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.030122    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9e9addb-0874-4848-bab8-325d88f63369" containerName="cleanup"
Sep 09 21:00:10.030147 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.030129    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9e9addb-0874-4848-bab8-325d88f63369" containerName="cleanup"
Sep 09 21:00:10.030239 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.030193    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="099ebc12-123b-43d9-8208-f661340dcadd" containerName="cleanup"
Sep 09 21:00:10.030239 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.030202    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9e9addb-0874-4848-bab8-325d88f63369" containerName="cleanup"
Sep 09 21:00:10.033092 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.033074    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:10.037150 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.037130    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-06e1ff\""
Sep 09 21:00:10.037410 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.037389    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-06e1ff-dockercfg-kkjc7\""
Sep 09 21:00:10.051197 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.051174    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"]
Sep 09 21:00:10.184243 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.184210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4f53bb-bf35-4a43-adc4-af50067f87bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs\" (UID: \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:10.184426 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.184270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsk4g\" (UniqueName: \"kubernetes.io/projected/5b4f53bb-bf35-4a43-adc4-af50067f87bf-kube-api-access-bsk4g\") pod \"payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs\" (UID: \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:10.285383 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.285355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4f53bb-bf35-4a43-adc4-af50067f87bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs\" (UID: \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:10.285748 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.285438    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsk4g\" (UniqueName: \"kubernetes.io/projected/5b4f53bb-bf35-4a43-adc4-af50067f87bf-kube-api-access-bsk4g\") pod \"payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs\" (UID: \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:10.286128 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.286109    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4f53bb-bf35-4a43-adc4-af50067f87bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs\" (UID: \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:10.294483 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.294459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsk4g\" (UniqueName: \"kubernetes.io/projected/5b4f53bb-bf35-4a43-adc4-af50067f87bf-kube-api-access-bsk4g\") pod \"payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs\" (UID: \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:10.343446 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.343383    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:10.616342 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:10.616304    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"]
Sep 09 21:00:10.618302 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:00:10.618271    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b4f53bb_bf35_4a43_adc4_af50067f87bf.slice/crio-e9409a46c88f0dd04d25f3f35fc731d846afc8dd8f3c439c6f5468b2d24946a8 WatchSource:0}: Error finding container e9409a46c88f0dd04d25f3f35fc731d846afc8dd8f3c439c6f5468b2d24946a8: Status 404 returned error can't find the container with id e9409a46c88f0dd04d25f3f35fc731d846afc8dd8f3c439c6f5468b2d24946a8
Sep 09 21:00:11.448386 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:11.448343    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs" event={"ID":"5b4f53bb-bf35-4a43-adc4-af50067f87bf","Type":"ContainerStarted","Data":"6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651"}
Sep 09 21:00:11.448764 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:11.448439    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs" event={"ID":"5b4f53bb-bf35-4a43-adc4-af50067f87bf","Type":"ContainerStarted","Data":"e9409a46c88f0dd04d25f3f35fc731d846afc8dd8f3c439c6f5468b2d24946a8"}
Sep 09 21:00:11.448764 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:11.448471    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:11.467887 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:11.467827    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs" podStartSLOduration=1.467810171 podStartE2EDuration="1.467810171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:11.467361777 +0000 UTC m=+1191.659199375" watchObservedRunningTime="2026-09-09 21:00:11.467810171 +0000 UTC m=+1191.659647772"
Sep 09 21:00:20.165703 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.165668    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"]
Sep 09 21:00:20.169247 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.169228    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.172559 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.172538    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e7c6aa\"/\"default-dockercfg-4lbsd\""
Sep 09 21:00:20.172674 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.172559    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e7c6aa\"/\"kube-root-ca.crt\""
Sep 09 21:00:20.172674 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.172563    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e7c6aa\"/\"openshift-service-ca.crt\""
Sep 09 21:00:20.172674 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.172537    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e7c6aa\"/\"noten-model-e7c6aa-kserve-self-signed-certs\""
Sep 09 21:00:20.178993 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.178953    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:20.180332 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.180299    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"]
Sep 09 21:00:20.273596 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.273568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-dshm\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.273741 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.273601    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-tmp-dir\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.273741 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.273637    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/8fc94da6-97ab-443c-87d5-439e78e54cbb-kube-api-access-hrr9q\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.273741 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.273695    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-home\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.273741 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.273720    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-model-cache\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.273741 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.273738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc94da6-97ab-443c-87d5-439e78e54cbb-tls-certs\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.374819 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.374786    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-home\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.375001 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.374836    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-model-cache\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.375001 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.374865    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc94da6-97ab-443c-87d5-439e78e54cbb-tls-certs\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.375001 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.374979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-dshm\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.375186 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.375019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-tmp-dir\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.375186 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.375108    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/8fc94da6-97ab-443c-87d5-439e78e54cbb-kube-api-access-hrr9q\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.375291 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.375215    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-home\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.375291 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.375224    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-model-cache\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.375593 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.375574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-tmp-dir\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.377445 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.377426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-dshm\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.377895 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.377879    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e7c6aa\"/\"noten-model-e7c6aa-kserve-self-signed-certs\""
Sep 09 21:00:20.384926 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.384898    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e7c6aa\"/\"kube-root-ca.crt\""
Sep 09 21:00:20.387886 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.387868    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc94da6-97ab-443c-87d5-439e78e54cbb-tls-certs\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.396338 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.396299    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e7c6aa\"/\"openshift-service-ca.crt\""
Sep 09 21:00:20.406228 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.406207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/8fc94da6-97ab-443c-87d5-439e78e54cbb-kube-api-access-hrr9q\") pod \"noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") " pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.474238 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.474168    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:20.481667 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.481647    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e7c6aa\"/\"default-dockercfg-4lbsd\""
Sep 09 21:00:20.490044 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.490021    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:20.762539 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:20.762515    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"]
Sep 09 21:00:20.764146 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:00:20.764123    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fc94da6_97ab_443c_87d5_439e78e54cbb.slice/crio-3195ba5c8c47f50918fcf4886e5094f2c69a0cc8603d619d8a67b3ec22d935c7 WatchSource:0}: Error finding container 3195ba5c8c47f50918fcf4886e5094f2c69a0cc8603d619d8a67b3ec22d935c7: Status 404 returned error can't find the container with id 3195ba5c8c47f50918fcf4886e5094f2c69a0cc8603d619d8a67b3ec22d935c7
Sep 09 21:00:21.499914 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:21.499878    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr" event={"ID":"8fc94da6-97ab-443c-87d5-439e78e54cbb","Type":"ContainerStarted","Data":"0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b"}
Sep 09 21:00:21.499914 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:21.499919    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr" event={"ID":"8fc94da6-97ab-443c-87d5-439e78e54cbb","Type":"ContainerStarted","Data":"3195ba5c8c47f50918fcf4886e5094f2c69a0cc8603d619d8a67b3ec22d935c7"}
Sep 09 21:00:21.520612 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:21.520559    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr" podStartSLOduration=1.520545808 podStartE2EDuration="1.520545808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:21.517975111 +0000 UTC m=+1201.709812709" watchObservedRunningTime="2026-09-09 21:00:21.520545808 +0000 UTC m=+1201.712383407"
Sep 09 21:00:22.089913 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:22.089869    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:22.455506 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:22.455434    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:00:30.284096 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:30.284061    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:30.490329 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:30.490275    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:30.490329 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:30.490335    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:30.502748 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:30.502720    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:30.560127 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:30.560064    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:00:31.793674 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:31.793642    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:32.038214 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.038183    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:32.058971 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.058901    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"]
Sep 09 21:00:32.059140 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.059116    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl" podUID="645b35e8-81a5-4083-9ed9-87b218e01fc2" containerName="payload-processing" containerID="cri-o://e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37" gracePeriod=30
Sep 09 21:00:32.069819 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.069791    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"]
Sep 09 21:00:32.070203 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.070158    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk" podUID="d32f1dac-f4ea-4c7b-a824-65316d76dfb6" containerName="payload-pre-processing" containerID="cri-o://93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457" gracePeriod=30
Sep 09 21:00:32.136412 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.136373    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:32.235734 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.235698    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:32.323475 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.323415    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 21:00:32.334436 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.334408    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:32.354271 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.354249    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 21:00:32.378956 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.378933    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-plugins-config-volume\") pod \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\" (UID: \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\") "
Sep 09 21:00:32.379077 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.378979    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b35e8-81a5-4083-9ed9-87b218e01fc2-plugins-config-volume\") pod \"645b35e8-81a5-4083-9ed9-87b218e01fc2\" (UID: \"645b35e8-81a5-4083-9ed9-87b218e01fc2\") "
Sep 09 21:00:32.379077 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.378995    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jw6rk\" (UniqueName: \"kubernetes.io/projected/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-kube-api-access-jw6rk\") pod \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\" (UID: \"d32f1dac-f4ea-4c7b-a824-65316d76dfb6\") "
Sep 09 21:00:32.379153 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.379076    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km8nl\" (UniqueName: \"kubernetes.io/projected/645b35e8-81a5-4083-9ed9-87b218e01fc2-kube-api-access-km8nl\") pod \"645b35e8-81a5-4083-9ed9-87b218e01fc2\" (UID: \"645b35e8-81a5-4083-9ed9-87b218e01fc2\") "
Sep 09 21:00:32.379581 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.379327    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d32f1dac-f4ea-4c7b-a824-65316d76dfb6" (UID: "d32f1dac-f4ea-4c7b-a824-65316d76dfb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:32.379581 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.379379    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/645b35e8-81a5-4083-9ed9-87b218e01fc2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "645b35e8-81a5-4083-9ed9-87b218e01fc2" (UID: "645b35e8-81a5-4083-9ed9-87b218e01fc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:32.379581 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.379515    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:32.379581 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.379534    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/645b35e8-81a5-4083-9ed9-87b218e01fc2-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:32.381114 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.381088    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-kube-api-access-jw6rk" (OuterVolumeSpecName: "kube-api-access-jw6rk") pod "d32f1dac-f4ea-4c7b-a824-65316d76dfb6" (UID: "d32f1dac-f4ea-4c7b-a824-65316d76dfb6"). InnerVolumeSpecName "kube-api-access-jw6rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:32.381205 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.381156    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/645b35e8-81a5-4083-9ed9-87b218e01fc2-kube-api-access-km8nl" (OuterVolumeSpecName: "kube-api-access-km8nl") pod "645b35e8-81a5-4083-9ed9-87b218e01fc2" (UID: "645b35e8-81a5-4083-9ed9-87b218e01fc2"). InnerVolumeSpecName "kube-api-access-km8nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:32.480888 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.480854    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw6rk\" (UniqueName: \"kubernetes.io/projected/d32f1dac-f4ea-4c7b-a824-65316d76dfb6-kube-api-access-jw6rk\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:32.480888 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.480880    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km8nl\" (UniqueName: \"kubernetes.io/projected/645b35e8-81a5-4083-9ed9-87b218e01fc2-kube-api-access-km8nl\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:32.560246 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.560212    2566 generic.go:356] "Generic (PLEG): container finished" podID="645b35e8-81a5-4083-9ed9-87b218e01fc2" containerID="e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37" exitCode=0
Sep 09 21:00:32.560382 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.560261    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"
Sep 09 21:00:32.560382 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.560294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl" event={"ID":"645b35e8-81a5-4083-9ed9-87b218e01fc2","Type":"ContainerDied","Data":"e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37"}
Sep 09 21:00:32.560382 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.560361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl" event={"ID":"645b35e8-81a5-4083-9ed9-87b218e01fc2","Type":"ContainerDied","Data":"61236b30b21cecf45b9928dcdca1f097b9a09cf5156385cdda18db89e1b69acb"}
Sep 09 21:00:32.560554 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.560383    2566 scope.go:117] "RemoveContainer" containerID="e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37"
Sep 09 21:00:32.561835 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.561815    2566 generic.go:356] "Generic (PLEG): container finished" podID="d32f1dac-f4ea-4c7b-a824-65316d76dfb6" containerID="93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457" exitCode=0
Sep 09 21:00:32.561941 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.561862    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"
Sep 09 21:00:32.561998 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.561863    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk" event={"ID":"d32f1dac-f4ea-4c7b-a824-65316d76dfb6","Type":"ContainerDied","Data":"93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457"}
Sep 09 21:00:32.562041 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.561997    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk" event={"ID":"d32f1dac-f4ea-4c7b-a824-65316d76dfb6","Type":"ContainerDied","Data":"2cfabaf4cdca49fea12ebdfe7a269220658bc3166cea928cb252f11e2b338e48"}
Sep 09 21:00:32.570198 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.570174    2566 scope.go:117] "RemoveContainer" containerID="e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37"
Sep 09 21:00:32.570496 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:00:32.570476    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37\": container with ID starting with e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37 not found: ID does not exist" containerID="e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37"
Sep 09 21:00:32.570572 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.570503    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37"} err="failed to get container status \"e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37\": rpc error: code = NotFound desc = could not find container \"e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37\": container with ID starting with e12e3e36f0b7c9ba6ff665f84101992b954c855bed30232650324a7d6ddbdd37 not found: ID does not exist"
Sep 09 21:00:32.570572 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.570523    2566 scope.go:117] "RemoveContainer" containerID="93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457"
Sep 09 21:00:32.578267 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.578249    2566 scope.go:117] "RemoveContainer" containerID="93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457"
Sep 09 21:00:32.578522 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:00:32.578505    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457\": container with ID starting with 93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457 not found: ID does not exist" containerID="93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457"
Sep 09 21:00:32.578584 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.578527    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457"} err="failed to get container status \"93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457\": rpc error: code = NotFound desc = could not find container \"93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457\": container with ID starting with 93522bffc04a423c51b153cd582340cd1a72985b8dde78c1e6ecafad7258b457 not found: ID does not exist"
Sep 09 21:00:32.585861 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.585841    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"]
Sep 09 21:00:32.591733 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.591714    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5eef90-546c7855bf-ptrcl"]
Sep 09 21:00:32.597978 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.597954    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:32.603203 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.603183    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"]
Sep 09 21:00:32.607669 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:32.607645    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-5eef90-5cf6d7d898-fkdbk"]
Sep 09 21:00:33.534154 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:33.534123    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:34.287041 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:34.287010    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="645b35e8-81a5-4083-9ed9-87b218e01fc2" path="/var/lib/kubelet/pods/645b35e8-81a5-4083-9ed9-87b218e01fc2/volumes"
Sep 09 21:00:34.287397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:34.287383    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d32f1dac-f4ea-4c7b-a824-65316d76dfb6" path="/var/lib/kubelet/pods/d32f1dac-f4ea-4c7b-a824-65316d76dfb6/volumes"
Sep 09 21:00:35.864118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:35.864089    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:00:38.747896 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:38.747855    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:38.997210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:38.997178    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"]
Sep 09 21:00:38.997451 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:38.997429    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy" containerID="cri-o://e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190" gracePeriod=30
Sep 09 21:00:40.001030 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:40.000987    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:40.100750 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:40.100716    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:40.269030 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:40.268952    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"]
Sep 09 21:00:40.288957 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:40.288914    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:42.650077 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.650042    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:42.912679 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.912591    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"]
Sep 09 21:00:42.912987 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.912973    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d32f1dac-f4ea-4c7b-a824-65316d76dfb6" containerName="payload-pre-processing"
Sep 09 21:00:42.912987 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.912987    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d32f1dac-f4ea-4c7b-a824-65316d76dfb6" containerName="payload-pre-processing"
Sep 09 21:00:42.913114 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.913009    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="645b35e8-81a5-4083-9ed9-87b218e01fc2" containerName="payload-processing"
Sep 09 21:00:42.913114 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.913015    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="645b35e8-81a5-4083-9ed9-87b218e01fc2" containerName="payload-processing"
Sep 09 21:00:42.913114 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.913076    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d32f1dac-f4ea-4c7b-a824-65316d76dfb6" containerName="payload-pre-processing"
Sep 09 21:00:42.913114 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.913082    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="645b35e8-81a5-4083-9ed9-87b218e01fc2" containerName="payload-processing"
Sep 09 21:00:42.920233 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.920212    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:42.923221 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.923200    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-12f2a92f-openshift-default-dockercfg-xtrnx\""
Sep 09 21:00:42.930021 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.930002    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"]
Sep 09 21:00:42.976190 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.976153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-socket\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:42.976295 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.976198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-token\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:42.976295 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.976262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-envoy\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:42.976406 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.976340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-certs\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:42.976406 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.976359    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-data\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:42.976406 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.976385    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-credential-socket\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:42.976406 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.976401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istiod-ca-cert\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:42.976532 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.976483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-podinfo\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:42.976532 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:42.976514    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbzdq\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-kube-api-access-kbzdq\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-token\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077429 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-envoy\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077429 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-certs\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077429 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077290    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-data\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077429 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077326    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-credential-socket\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077429 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077361    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istiod-ca-cert\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077429 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077434    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-podinfo\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077750 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077463    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbzdq\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-kube-api-access-kbzdq\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077750 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-socket\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.077750 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.077726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-certs\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.078079 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.078052    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-credential-socket\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.078178 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.078056    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-data\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.078178 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.078144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-socket\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.078552 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.078529    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istiod-ca-cert\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.079831 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.079811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-envoy\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.079910 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.079893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-podinfo\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.085917 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.085891    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-token\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.086265 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.086243    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbzdq\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-kube-api-access-kbzdq\") pod \"e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") " pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.231329 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.231273    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:43.507429 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.507403    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"]
Sep 09 21:00:43.508718 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:00:43.508694    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5e1ec7_a190_4397_b8ff_cc4ec236340e.slice/crio-6e1f0ed37ef540d878c94848a08cf7ca43d52df69575a2c3c41c86a7aff56d10 WatchSource:0}: Error finding container 6e1f0ed37ef540d878c94848a08cf7ca43d52df69575a2c3c41c86a7aff56d10: Status 404 returned error can't find the container with id 6e1f0ed37ef540d878c94848a08cf7ca43d52df69575a2c3c41c86a7aff56d10
Sep 09 21:00:43.510966 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.510936    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:00:43.511047 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.510999    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:00:43.511047 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.511026    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:00:43.622626 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.622586    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" event={"ID":"7b5e1ec7-a190-4397-b8ff-cc4ec236340e","Type":"ContainerStarted","Data":"6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21"}
Sep 09 21:00:43.622810 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.622661    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" event={"ID":"7b5e1ec7-a190-4397-b8ff-cc4ec236340e","Type":"ContainerStarted","Data":"6e1f0ed37ef540d878c94848a08cf7ca43d52df69575a2c3c41c86a7aff56d10"}
Sep 09 21:00:43.644897 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.644844    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" podStartSLOduration=1.644827788 podStartE2EDuration="1.644827788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:43.643794834 +0000 UTC m=+1223.835632434" watchObservedRunningTime="2026-09-09 21:00:43.644827788 +0000 UTC m=+1223.836665386"
Sep 09 21:00:43.859882 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.859802    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:43.959931 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:43.959893    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:44.231709 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.231682    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:44.237028 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.237008    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:44.249352 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.249332    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 21:00:44.290057 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290031    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-credential-socket\") pod \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") "
Sep 09 21:00:44.290180 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290070    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-podinfo\") pod \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") "
Sep 09 21:00:44.290180 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290100    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fgpq\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-kube-api-access-9fgpq\") pod \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") "
Sep 09 21:00:44.290180 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290137    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istiod-ca-cert\") pod \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") "
Sep 09 21:00:44.290180 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290163    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-token\") pod \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") "
Sep 09 21:00:44.290440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290232    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-certs\") pod \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") "
Sep 09 21:00:44.290440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290269    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-socket\") pod \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") "
Sep 09 21:00:44.290440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290292    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ffa8e0d1-e6d1-4678-b4c6-c99a96981487" (UID: "ffa8e0d1-e6d1-4678-b4c6-c99a96981487"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:44.290440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290360    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-envoy\") pod \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") "
Sep 09 21:00:44.290440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290401    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-data\") pod \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\" (UID: \"ffa8e0d1-e6d1-4678-b4c6-c99a96981487\") "
Sep 09 21:00:44.290701 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290558    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ffa8e0d1-e6d1-4678-b4c6-c99a96981487" (UID: "ffa8e0d1-e6d1-4678-b4c6-c99a96981487"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:44.290701 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290594    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ffa8e0d1-e6d1-4678-b4c6-c99a96981487" (UID: "ffa8e0d1-e6d1-4678-b4c6-c99a96981487"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:44.290803 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290763    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-data" (OuterVolumeSpecName: "istio-data") pod "ffa8e0d1-e6d1-4678-b4c6-c99a96981487" (UID: "ffa8e0d1-e6d1-4678-b4c6-c99a96981487"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:44.290803 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290783    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ffa8e0d1-e6d1-4678-b4c6-c99a96981487" (UID: "ffa8e0d1-e6d1-4678-b4c6-c99a96981487"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:44.290967 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290944    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:44.291082 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290970    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-workload-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:44.291082 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290985    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-data\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:44.291082 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.290998    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-credential-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:44.291082 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.291011    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istiod-ca-cert\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:44.292509 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.292456    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-token" (OuterVolumeSpecName: "istio-token") pod "ffa8e0d1-e6d1-4678-b4c6-c99a96981487" (UID: "ffa8e0d1-e6d1-4678-b4c6-c99a96981487"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:44.292509 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.292459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ffa8e0d1-e6d1-4678-b4c6-c99a96981487" (UID: "ffa8e0d1-e6d1-4678-b4c6-c99a96981487"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:00:44.293002 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.292982    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-kube-api-access-9fgpq" (OuterVolumeSpecName: "kube-api-access-9fgpq") pod "ffa8e0d1-e6d1-4678-b4c6-c99a96981487" (UID: "ffa8e0d1-e6d1-4678-b4c6-c99a96981487"). InnerVolumeSpecName "kube-api-access-9fgpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:44.293069 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.293040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ffa8e0d1-e6d1-4678-b4c6-c99a96981487" (UID: "ffa8e0d1-e6d1-4678-b4c6-c99a96981487"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:44.392257 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.392200    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-podinfo\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:44.392257 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.392224    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fgpq\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-kube-api-access-9fgpq\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:44.392257 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.392235    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-token\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:44.392257 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.392243    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffa8e0d1-e6d1-4678-b4c6-c99a96981487-istio-envoy\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:44.630509 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.630476    2566 generic.go:356] "Generic (PLEG): container finished" podID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerID="e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190" exitCode=0
Sep 09 21:00:44.630638 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.630524    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"
Sep 09 21:00:44.630638 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.630519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" event={"ID":"ffa8e0d1-e6d1-4678-b4c6-c99a96981487","Type":"ContainerDied","Data":"e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190"}
Sep 09 21:00:44.630638 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.630636    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c" event={"ID":"ffa8e0d1-e6d1-4678-b4c6-c99a96981487","Type":"ContainerDied","Data":"cdfec216f2e6ff67a018300c9430f76afa477087b8afb2db194f64342f755c42"}
Sep 09 21:00:44.630793 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.630661    2566 scope.go:117] "RemoveContainer" containerID="e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190"
Sep 09 21:00:44.631009 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.630985    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:44.632150 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.632026    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:00:44.642145 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.641460    2566 scope.go:117] "RemoveContainer" containerID="9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6"
Sep 09 21:00:44.650340 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.650299    2566 scope.go:117] "RemoveContainer" containerID="e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190"
Sep 09 21:00:44.650606 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:00:44.650590    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190\": container with ID starting with e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190 not found: ID does not exist" containerID="e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190"
Sep 09 21:00:44.650647 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.650613    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190"} err="failed to get container status \"e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190\": rpc error: code = NotFound desc = could not find container \"e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190\": container with ID starting with e62ebf5ba00c3542cf8288ad0068df3229b80ecd68745af1cef3665f3702f190 not found: ID does not exist"
Sep 09 21:00:44.650647 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.650628    2566 scope.go:117] "RemoveContainer" containerID="9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6"
Sep 09 21:00:44.650861 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:00:44.650842    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6\": container with ID starting with 9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6 not found: ID does not exist" containerID="9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6"
Sep 09 21:00:44.650912 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.650870    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6"} err="failed to get container status \"9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6\": rpc error: code = NotFound desc = could not find container \"9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6\": container with ID starting with 9595b8b86b61d1a096d058f03bddd8b42e14be21f0dd45eed12070eb1a43c2c6 not found: ID does not exist"
Sep 09 21:00:44.669432 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.669406    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"]
Sep 09 21:00:44.676842 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:44.676821    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-5eef90-openshift-default-5545bdc57d-vb22c"]
Sep 09 21:00:45.566194 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:45.566153    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:45.690574 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:45.690548    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:46.287987 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:46.287958    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" path="/var/lib/kubelet/pods/ffa8e0d1-e6d1-4678-b4c6-c99a96981487/volumes"
Sep 09 21:00:47.515333 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.515281    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:47.615303 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.615272    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:47.660551 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.660522    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"]
Sep 09 21:00:47.660902 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.660890    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy"
Sep 09 21:00:47.660948 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.660903    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy"
Sep 09 21:00:47.660948 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.660915    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy"
Sep 09 21:00:47.660948 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.660920    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy"
Sep 09 21:00:47.661055 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.660981    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy"
Sep 09 21:00:47.661055 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.660989    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffa8e0d1-e6d1-4678-b4c6-c99a96981487" containerName="istio-proxy"
Sep 09 21:00:47.670613 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.670586    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:47.673166 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.673135    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-12f2a92f\""
Sep 09 21:00:47.673343 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.673244    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-12f2a92f-dockercfg-t6k4z\""
Sep 09 21:00:47.679743 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.679719    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"]
Sep 09 21:00:47.718823 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.718786    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:47.723001 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.722976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922\" (UID: \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:47.723121 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.723106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pll79\" (UniqueName: \"kubernetes.io/projected/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-kube-api-access-pll79\") pod \"payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922\" (UID: \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:47.804396 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.804300    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"]
Sep 09 21:00:47.807873 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.807852    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:47.821357 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.821331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"]
Sep 09 21:00:47.824199 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.824176    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pll79\" (UniqueName: \"kubernetes.io/projected/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-kube-api-access-pll79\") pod \"payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922\" (UID: \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:47.824335 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.824263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922\" (UID: \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:47.824940 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.824919    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922\" (UID: \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:47.837045 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.837026    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pll79\" (UniqueName: \"kubernetes.io/projected/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-kube-api-access-pll79\") pod \"payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922\" (UID: \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:47.925505 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.925476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bae4a8-74ba-471f-8001-a109941dcc0e-plugins-config-volume\") pod \"payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv\" (UID: \"58bae4a8-74ba-471f-8001-a109941dcc0e\") " pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:47.925695 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.925668    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zzh\" (UniqueName: \"kubernetes.io/projected/58bae4a8-74ba-471f-8001-a109941dcc0e-kube-api-access-79zzh\") pod \"payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv\" (UID: \"58bae4a8-74ba-471f-8001-a109941dcc0e\") " pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:47.981839 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:47.981816    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:48.026518 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.026484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zzh\" (UniqueName: \"kubernetes.io/projected/58bae4a8-74ba-471f-8001-a109941dcc0e-kube-api-access-79zzh\") pod \"payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv\" (UID: \"58bae4a8-74ba-471f-8001-a109941dcc0e\") " pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:48.026518 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.026535    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bae4a8-74ba-471f-8001-a109941dcc0e-plugins-config-volume\") pod \"payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv\" (UID: \"58bae4a8-74ba-471f-8001-a109941dcc0e\") " pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:48.027226 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.027207    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bae4a8-74ba-471f-8001-a109941dcc0e-plugins-config-volume\") pod \"payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv\" (UID: \"58bae4a8-74ba-471f-8001-a109941dcc0e\") " pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:48.035956 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.035931    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zzh\" (UniqueName: \"kubernetes.io/projected/58bae4a8-74ba-471f-8001-a109941dcc0e-kube-api-access-79zzh\") pod \"payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv\" (UID: \"58bae4a8-74ba-471f-8001-a109941dcc0e\") " pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:48.118122 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.118052    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:48.259410 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.259381    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"]
Sep 09 21:00:48.261073 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:00:48.261046    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1be8542_489a_4ed3_9b0e_084a2e3d5e8e.slice/crio-7137d51ae80024ea71851f22c470d203c8273b92f934a5bce3491af99e2ab486 WatchSource:0}: Error finding container 7137d51ae80024ea71851f22c470d203c8273b92f934a5bce3491af99e2ab486: Status 404 returned error can't find the container with id 7137d51ae80024ea71851f22c470d203c8273b92f934a5bce3491af99e2ab486
Sep 09 21:00:48.401875 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.401851    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"]
Sep 09 21:00:48.403976 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:00:48.403953    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58bae4a8_74ba_471f_8001_a109941dcc0e.slice/crio-be6344117ad4306cc6ee3982b6d51fae7356d07fac21a277968cb2d0e063c7d6 WatchSource:0}: Error finding container be6344117ad4306cc6ee3982b6d51fae7356d07fac21a277968cb2d0e063c7d6: Status 404 returned error can't find the container with id be6344117ad4306cc6ee3982b6d51fae7356d07fac21a277968cb2d0e063c7d6
Sep 09 21:00:48.656186 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.656090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922" event={"ID":"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e","Type":"ContainerStarted","Data":"21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39"}
Sep 09 21:00:48.656186 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.656152    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922" event={"ID":"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e","Type":"ContainerStarted","Data":"7137d51ae80024ea71851f22c470d203c8273b92f934a5bce3491af99e2ab486"}
Sep 09 21:00:48.656681 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.656276    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:48.657906 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.657881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv" event={"ID":"58bae4a8-74ba-471f-8001-a109941dcc0e","Type":"ContainerStarted","Data":"246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80"}
Sep 09 21:00:48.658011 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.657935    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:48.658011 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.657956    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv" event={"ID":"58bae4a8-74ba-471f-8001-a109941dcc0e","Type":"ContainerStarted","Data":"be6344117ad4306cc6ee3982b6d51fae7356d07fac21a277968cb2d0e063c7d6"}
Sep 09 21:00:48.679883 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.679838    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922" podStartSLOduration=1.679825095 podStartE2EDuration="1.679825095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:48.677967128 +0000 UTC m=+1228.869804727" watchObservedRunningTime="2026-09-09 21:00:48.679825095 +0000 UTC m=+1228.871662692"
Sep 09 21:00:48.701007 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:48.700967    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv" podStartSLOduration=1.700950643 podStartE2EDuration="1.700950643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:48.700202176 +0000 UTC m=+1228.892039780" watchObservedRunningTime="2026-09-09 21:00:48.700950643 +0000 UTC m=+1228.892788242"
Sep 09 21:00:49.666525 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:49.666484    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:50.305945 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.305901    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:50.340559 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.340520    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"]
Sep 09 21:00:50.340782 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.340752    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv" podUID="223badda-803d-469f-885b-2091d943bbf9" containerName="payload-pre-processing" containerID="cri-o://4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4" gracePeriod=30
Sep 09 21:00:50.406031 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.405997    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:50.506014 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.505971    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:50.590979 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.590958    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 21:00:50.607133 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.607100    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:50.650827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.650790    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223badda-803d-469f-885b-2091d943bbf9-plugins-config-volume\") pod \"223badda-803d-469f-885b-2091d943bbf9\" (UID: \"223badda-803d-469f-885b-2091d943bbf9\") "
Sep 09 21:00:50.650995 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.650955    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjbzq\" (UniqueName: \"kubernetes.io/projected/223badda-803d-469f-885b-2091d943bbf9-kube-api-access-bjbzq\") pod \"223badda-803d-469f-885b-2091d943bbf9\" (UID: \"223badda-803d-469f-885b-2091d943bbf9\") "
Sep 09 21:00:50.651192 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.651170    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/223badda-803d-469f-885b-2091d943bbf9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "223badda-803d-469f-885b-2091d943bbf9" (UID: "223badda-803d-469f-885b-2091d943bbf9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:00:50.651364 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.651304    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/223badda-803d-469f-885b-2091d943bbf9-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:50.653191 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.653153    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/223badda-803d-469f-885b-2091d943bbf9-kube-api-access-bjbzq" (OuterVolumeSpecName: "kube-api-access-bjbzq") pod "223badda-803d-469f-885b-2091d943bbf9" (UID: "223badda-803d-469f-885b-2091d943bbf9"). InnerVolumeSpecName "kube-api-access-bjbzq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:50.670953 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.670926    2566 generic.go:356] "Generic (PLEG): container finished" podID="223badda-803d-469f-885b-2091d943bbf9" containerID="4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4" exitCode=0
Sep 09 21:00:50.671344 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.670967    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"
Sep 09 21:00:50.671344 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.671012    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv" event={"ID":"223badda-803d-469f-885b-2091d943bbf9","Type":"ContainerDied","Data":"4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4"}
Sep 09 21:00:50.671344 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.671111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv" event={"ID":"223badda-803d-469f-885b-2091d943bbf9","Type":"ContainerDied","Data":"d2bce6906761c2124d629ee92255448194eed60bfcdf0ed635cfb3a2a85375d7"}
Sep 09 21:00:50.671344 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.671146    2566 scope.go:117] "RemoveContainer" containerID="4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4"
Sep 09 21:00:50.680369 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.680351    2566 scope.go:117] "RemoveContainer" containerID="4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4"
Sep 09 21:00:50.680636 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:00:50.680617    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4\": container with ID starting with 4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4 not found: ID does not exist" containerID="4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4"
Sep 09 21:00:50.680718 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.680642    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4"} err="failed to get container status \"4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4\": rpc error: code = NotFound desc = could not find container \"4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4\": container with ID starting with 4b9bbdf17ca9e7d906d82a359a6d194d01067a571c6c7b6a7844179701f8bea4 not found: ID does not exist"
Sep 09 21:00:50.696194 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.696171    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"]
Sep 09 21:00:50.699544 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.699524    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-423e6a-6579667cfd-r9vmv"]
Sep 09 21:00:50.752013 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.751984    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjbzq\" (UniqueName: \"kubernetes.io/projected/223badda-803d-469f-885b-2091d943bbf9-kube-api-access-bjbzq\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:50.934195 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:50.934112    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:52.286625 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:52.286592    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="223badda-803d-469f-885b-2091d943bbf9" path="/var/lib/kubelet/pods/223badda-803d-469f-885b-2091d943bbf9/volumes"
Sep 09 21:00:53.028210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:53.028172    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:54.074122 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:54.074083    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:54.163398 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:54.163368    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:54.722403 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:54.722372    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:55.293110 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.293074    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5" podUID="f857a1dd-621f-4a07-b818-a277c7b9537b" containerName="main" containerID="cri-o://e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1" gracePeriod=45
Sep 09 21:00:55.548685 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.548635    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 21:00:55.693030 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693004    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f857a1dd-621f-4a07-b818-a277c7b9537b-tls-certs\") pod \"f857a1dd-621f-4a07-b818-a277c7b9537b\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") "
Sep 09 21:00:55.693210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693037    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-model-cache\") pod \"f857a1dd-621f-4a07-b818-a277c7b9537b\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") "
Sep 09 21:00:55.693210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693177    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-tmp-dir\") pod \"f857a1dd-621f-4a07-b818-a277c7b9537b\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") "
Sep 09 21:00:55.693380 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693246    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7vv6\" (UniqueName: \"kubernetes.io/projected/f857a1dd-621f-4a07-b818-a277c7b9537b-kube-api-access-j7vv6\") pod \"f857a1dd-621f-4a07-b818-a277c7b9537b\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") "
Sep 09 21:00:55.693380 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693298    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-home\") pod \"f857a1dd-621f-4a07-b818-a277c7b9537b\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") "
Sep 09 21:00:55.693380 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693350    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-dshm\") pod \"f857a1dd-621f-4a07-b818-a277c7b9537b\" (UID: \"f857a1dd-621f-4a07-b818-a277c7b9537b\") "
Sep 09 21:00:55.693540 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f857a1dd-621f-4a07-b818-a277c7b9537b" (UID: "f857a1dd-621f-4a07-b818-a277c7b9537b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:55.693540 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693436    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f857a1dd-621f-4a07-b818-a277c7b9537b" (UID: "f857a1dd-621f-4a07-b818-a277c7b9537b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:55.693540 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693517    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-home" (OuterVolumeSpecName: "home") pod "f857a1dd-621f-4a07-b818-a277c7b9537b" (UID: "f857a1dd-621f-4a07-b818-a277c7b9537b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:55.693722 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693703    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:55.693768 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693730    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:55.693768 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.693744    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:55.703484 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.703440    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f857a1dd-621f-4a07-b818-a277c7b9537b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f857a1dd-621f-4a07-b818-a277c7b9537b" (UID: "f857a1dd-621f-4a07-b818-a277c7b9537b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:00:55.703484 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.703458    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f857a1dd-621f-4a07-b818-a277c7b9537b-kube-api-access-j7vv6" (OuterVolumeSpecName: "kube-api-access-j7vv6") pod "f857a1dd-621f-4a07-b818-a277c7b9537b" (UID: "f857a1dd-621f-4a07-b818-a277c7b9537b"). InnerVolumeSpecName "kube-api-access-j7vv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:00:55.703677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.703487    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-dshm" (OuterVolumeSpecName: "dshm") pod "f857a1dd-621f-4a07-b818-a277c7b9537b" (UID: "f857a1dd-621f-4a07-b818-a277c7b9537b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:00:55.706335 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.706288    2566 generic.go:356] "Generic (PLEG): container finished" podID="f857a1dd-621f-4a07-b818-a277c7b9537b" containerID="e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1" exitCode=0
Sep 09 21:00:55.706456 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.706340    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5" event={"ID":"f857a1dd-621f-4a07-b818-a277c7b9537b","Type":"ContainerDied","Data":"e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1"}
Sep 09 21:00:55.706456 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.706368    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"
Sep 09 21:00:55.706456 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.706381    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5" event={"ID":"f857a1dd-621f-4a07-b818-a277c7b9537b","Type":"ContainerDied","Data":"3dec0688b992c3cf3874ff975b0c4783056a70383428f3288f2cf6fa4a2c49b3"}
Sep 09 21:00:55.706456 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.706399    2566 scope.go:117] "RemoveContainer" containerID="e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1"
Sep 09 21:00:55.717229 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.717211    2566 scope.go:117] "RemoveContainer" containerID="e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1"
Sep 09 21:00:55.717512 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:00:55.717495    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1\": container with ID starting with e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1 not found: ID does not exist" containerID="e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1"
Sep 09 21:00:55.717581 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.717519    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1"} err="failed to get container status \"e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1\": rpc error: code = NotFound desc = could not find container \"e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1\": container with ID starting with e20f15462485ae485f37b3eb89238bb7e7e752c3978e847938eedb3d8ac1a8d1 not found: ID does not exist"
Sep 09 21:00:55.730822 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.730801    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"]
Sep 09 21:00:55.734948 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.734930    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-423e6a/test-model-423e6a-kserve-7dcff7db75-cmvd5"]
Sep 09 21:00:55.794818 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.794797    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f857a1dd-621f-4a07-b818-a277c7b9537b-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:55.794818 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.794815    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f857a1dd-621f-4a07-b818-a277c7b9537b-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:55.794964 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:55.794824    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7vv6\" (UniqueName: \"kubernetes.io/projected/f857a1dd-621f-4a07-b818-a277c7b9537b-kube-api-access-j7vv6\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:00:56.287527 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:56.287499    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f857a1dd-621f-4a07-b818-a277c7b9537b" path="/var/lib/kubelet/pods/f857a1dd-621f-4a07-b818-a277c7b9537b/volumes"
Sep 09 21:00:56.462397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:56.462363    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:57.831223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.831194    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"]
Sep 09 21:00:57.831595 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.831580    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="223badda-803d-469f-885b-2091d943bbf9" containerName="payload-pre-processing"
Sep 09 21:00:57.831595 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.831595    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="223badda-803d-469f-885b-2091d943bbf9" containerName="payload-pre-processing"
Sep 09 21:00:57.831682 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.831608    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f857a1dd-621f-4a07-b818-a277c7b9537b" containerName="main"
Sep 09 21:00:57.831682 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.831613    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f857a1dd-621f-4a07-b818-a277c7b9537b" containerName="main"
Sep 09 21:00:57.831755 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.831703    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="223badda-803d-469f-885b-2091d943bbf9" containerName="payload-pre-processing"
Sep 09 21:00:57.831755 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.831714    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f857a1dd-621f-4a07-b818-a277c7b9537b" containerName="main"
Sep 09 21:00:57.836233 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.836215    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:57.838641 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.838613    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-615c6f-openshift-default-dockercfg-vl6v8\""
Sep 09 21:00:57.848128 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.848107    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"]
Sep 09 21:00:57.910859 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.910835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-socket\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:57.910960 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.910866    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff74b49-3c3f-473a-a02c-6e72b5079028-istiod-ca-cert\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:57.910960 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.910885    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964bv\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-kube-api-access-964bv\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:57.910960 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.910929    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-token\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:57.911075 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.910972    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-certs\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:57.911075 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.911009    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-podinfo\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:57.911075 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.911035    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-envoy\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:57.911075 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.911055    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-data\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:57.911192 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:57.911076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-credential-socket\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.011696 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.011666    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-socket\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.011696 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.011699    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff74b49-3c3f-473a-a02c-6e72b5079028-istiod-ca-cert\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.011880 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.011719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964bv\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-kube-api-access-964bv\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.011880 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.011737    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-token\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.011880 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.011856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-certs\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.012025 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.011916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-podinfo\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.012025 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.011971    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-envoy\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.012025 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.011995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-data\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.012025 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.012022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-credential-socket\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.012226 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.012128    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-socket\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.012271 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.012217    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-certs\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.012337 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.012322    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-credential-socket\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.012470 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.012450    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-data\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.012554 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.012532    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff74b49-3c3f-473a-a02c-6e72b5079028-istiod-ca-cert\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.014172 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.014151    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-envoy\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.014405 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.014389    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-podinfo\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.020373 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.020354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-token\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.020484 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.020421    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-964bv\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-kube-api-access-964bv\") pod \"e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") " pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.146791 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.146726    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:58.273655 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.273629    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:00:58.312811 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.312783    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"]
Sep 09 21:00:58.429513 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.429433    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"]
Sep 09 21:00:58.431791 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:00:58.431764    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff74b49_3c3f_473a_a02c_6e72b5079028.slice/crio-21615d6a79ed97877fed6978d4293b236e605a2694e60b8c0134f31e77394390 WatchSource:0}: Error finding container 21615d6a79ed97877fed6978d4293b236e605a2694e60b8c0134f31e77394390: Status 404 returned error can't find the container with id 21615d6a79ed97877fed6978d4293b236e605a2694e60b8c0134f31e77394390
Sep 09 21:00:58.433983 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.433954    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:00:58.434085 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.434015    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:00:58.434085 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.434047    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:00:58.658353 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.658292    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:00:58.724947 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.724861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" event={"ID":"fff74b49-3c3f-473a-a02c-6e72b5079028","Type":"ContainerStarted","Data":"370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8"}
Sep 09 21:00:58.724947 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.724931    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" event={"ID":"fff74b49-3c3f-473a-a02c-6e72b5079028","Type":"ContainerStarted","Data":"21615d6a79ed97877fed6978d4293b236e605a2694e60b8c0134f31e77394390"}
Sep 09 21:00:58.747943 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:58.747887    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" podStartSLOduration=1.747866468 podStartE2EDuration="1.747866468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:00:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:00:58.746975536 +0000 UTC m=+1238.938813132" watchObservedRunningTime="2026-09-09 21:00:58.747866468 +0000 UTC m=+1238.939704067"
Sep 09 21:00:59.147671 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:59.147639    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:59.152754 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:59.152730    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:59.666392 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:59.666363    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:00:59.666556 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:59.666437    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:00:59.732137 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:59.732106    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:00:59.733051 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:00:59.733020    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:01:00.744730 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:00.744673    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:00.835232 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:00.835197    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:02.048725 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:02.048691    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:02.145001 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:02.144949    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:02.758852 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:02.758817    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:02.853531 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:02.853493    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:08.127554 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:08.127515    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:09.797976 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:09.797945    2566 generic.go:356] "Generic (PLEG): container finished" podID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerID="6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21" exitCode=0
Sep 09 21:01:09.798411 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:09.797998    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" event={"ID":"7b5e1ec7-a190-4397-b8ff-cc4ec236340e","Type":"ContainerDied","Data":"6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21"}
Sep 09 21:01:09.798475 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:09.798439    2566 scope.go:117] "RemoveContainer" containerID="6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21"
Sep 09 21:01:09.799694 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:09.799668    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:01:09.799782 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:09.799732    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:01:09.799782 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:09.799765    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:01:10.420854 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:10.420811    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:10.804781 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:10.804744    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" event={"ID":"7b5e1ec7-a190-4397-b8ff-cc4ec236340e","Type":"ContainerStarted","Data":"c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f"}
Sep 09 21:01:11.232299 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:11.232267    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:01:11.233476 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:11.233452    2566 patch_prober.go:28] interesting pod/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:01:11.233534 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:11.233507    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 21:01:12.232762 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:12.232719    2566 patch_prober.go:28] interesting pod/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:01:12.233409 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:12.233376    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 21:01:13.232673 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.232632    2566 patch_prober.go:28] interesting pod/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:01:13.232971 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.232930    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 21:01:13.339098 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.339056    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr" podUID="8fc94da6-97ab-443c-87d5-439e78e54cbb" containerName="main" containerID="cri-o://0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b" gracePeriod=45
Sep 09 21:01:13.614795 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.614771    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:01:13.666220 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.666187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-tmp-dir\") pod \"8fc94da6-97ab-443c-87d5-439e78e54cbb\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") "
Sep 09 21:01:13.666392 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.666251    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc94da6-97ab-443c-87d5-439e78e54cbb-tls-certs\") pod \"8fc94da6-97ab-443c-87d5-439e78e54cbb\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") "
Sep 09 21:01:13.666392 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.666358    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/8fc94da6-97ab-443c-87d5-439e78e54cbb-kube-api-access-hrr9q\") pod \"8fc94da6-97ab-443c-87d5-439e78e54cbb\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") "
Sep 09 21:01:13.666392 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.666386    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-dshm\") pod \"8fc94da6-97ab-443c-87d5-439e78e54cbb\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") "
Sep 09 21:01:13.666578 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.666416    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-model-cache\") pod \"8fc94da6-97ab-443c-87d5-439e78e54cbb\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") "
Sep 09 21:01:13.666578 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.666451    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-home\") pod \"8fc94da6-97ab-443c-87d5-439e78e54cbb\" (UID: \"8fc94da6-97ab-443c-87d5-439e78e54cbb\") "
Sep 09 21:01:13.666578 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.666494    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8fc94da6-97ab-443c-87d5-439e78e54cbb" (UID: "8fc94da6-97ab-443c-87d5-439e78e54cbb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:13.666765 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.666743    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:01:13.667037 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.667007    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-home" (OuterVolumeSpecName: "home") pod "8fc94da6-97ab-443c-87d5-439e78e54cbb" (UID: "8fc94da6-97ab-443c-87d5-439e78e54cbb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:13.667244 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.667220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-model-cache" (OuterVolumeSpecName: "model-cache") pod "8fc94da6-97ab-443c-87d5-439e78e54cbb" (UID: "8fc94da6-97ab-443c-87d5-439e78e54cbb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:13.668812 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.668792    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-dshm" (OuterVolumeSpecName: "dshm") pod "8fc94da6-97ab-443c-87d5-439e78e54cbb" (UID: "8fc94da6-97ab-443c-87d5-439e78e54cbb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:01:13.669042 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.669024    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc94da6-97ab-443c-87d5-439e78e54cbb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8fc94da6-97ab-443c-87d5-439e78e54cbb" (UID: "8fc94da6-97ab-443c-87d5-439e78e54cbb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:01:13.669107 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.669027    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc94da6-97ab-443c-87d5-439e78e54cbb-kube-api-access-hrr9q" (OuterVolumeSpecName: "kube-api-access-hrr9q") pod "8fc94da6-97ab-443c-87d5-439e78e54cbb" (UID: "8fc94da6-97ab-443c-87d5-439e78e54cbb"). InnerVolumeSpecName "kube-api-access-hrr9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:01:13.768225 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.768141    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrr9q\" (UniqueName: \"kubernetes.io/projected/8fc94da6-97ab-443c-87d5-439e78e54cbb-kube-api-access-hrr9q\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:01:13.768225 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.768172    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:01:13.768225 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.768181    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:01:13.768225 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.768192    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc94da6-97ab-443c-87d5-439e78e54cbb-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:01:13.768225 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.768204    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc94da6-97ab-443c-87d5-439e78e54cbb-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:01:13.820749 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.820717    2566 generic.go:356] "Generic (PLEG): container finished" podID="8fc94da6-97ab-443c-87d5-439e78e54cbb" containerID="0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b" exitCode=0
Sep 09 21:01:13.820749 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.820746    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr" event={"ID":"8fc94da6-97ab-443c-87d5-439e78e54cbb","Type":"ContainerDied","Data":"0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b"}
Sep 09 21:01:13.820999 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.820760    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"
Sep 09 21:01:13.820999 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.820784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr" event={"ID":"8fc94da6-97ab-443c-87d5-439e78e54cbb","Type":"ContainerDied","Data":"3195ba5c8c47f50918fcf4886e5094f2c69a0cc8603d619d8a67b3ec22d935c7"}
Sep 09 21:01:13.820999 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.820806    2566 scope.go:117] "RemoveContainer" containerID="0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b"
Sep 09 21:01:13.851501 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.851463    2566 scope.go:117] "RemoveContainer" containerID="0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b"
Sep 09 21:01:13.851986 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:01:13.851946    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b\": container with ID starting with 0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b not found: ID does not exist" containerID="0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b"
Sep 09 21:01:13.852150 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.852125    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b"} err="failed to get container status \"0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b\": rpc error: code = NotFound desc = could not find container \"0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b\": container with ID starting with 0de45cb1f5af070804401660351e2fb84971ffa4667166102c5a061e86211e0b not found: ID does not exist"
Sep 09 21:01:13.867970 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:01:13.867946    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fc94da6_97ab_443c_87d5_439e78e54cbb.slice\": RecentStats: unable to find data in memory cache]"
Sep 09 21:01:13.871014 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.870991    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"]
Sep 09 21:01:13.879895 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:13.879870    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-e7c6aa/noten-model-e7c6aa-kserve-6fbc6d7ddc-2cpqr"]
Sep 09 21:01:14.232495 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:14.232461    2566 patch_prober.go:28] interesting pod/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:01:14.232797 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:14.232767    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 21:01:14.286481 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:14.286450    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc94da6-97ab-443c-87d5-439e78e54cbb" path="/var/lib/kubelet/pods/8fc94da6-97ab-443c-87d5-439e78e54cbb/volumes"
Sep 09 21:01:14.633518 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:14.633412    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:01:15.232193 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:15.232146    2566 patch_prober.go:28] interesting pod/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:01:15.232413 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:15.232217    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 21:01:16.231815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:16.231771    2566 patch_prober.go:28] interesting pod/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:01:16.232430 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:16.232400    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 21:01:17.231808 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:17.231764    2566 patch_prober.go:28] interesting pod/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 09 21:01:17.232024 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:17.231853    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 09 21:01:18.248192 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:18.248157    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:01:18.249617 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:18.249595    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:01:22.226971 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:22.226926    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:23.119855 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:23.119818    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:24.379403 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:24.379331    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:01:24.892962 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:24.892926    2566 generic.go:356] "Generic (PLEG): container finished" podID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerID="370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8" exitCode=0
Sep 09 21:01:24.893152 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:24.893003    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" event={"ID":"fff74b49-3c3f-473a-a02c-6e72b5079028","Type":"ContainerDied","Data":"370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8"}
Sep 09 21:01:24.893449 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:24.893433    2566 scope.go:117] "RemoveContainer" containerID="370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8"
Sep 09 21:01:24.894627 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:24.894601    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:01:24.894711 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:24.894672    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:01:24.894711 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:24.894703    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:01:25.568761 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:25.568730    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:25.899672 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:25.899592    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" event={"ID":"fff74b49-3c3f-473a-a02c-6e72b5079028","Type":"ContainerStarted","Data":"db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e"}
Sep 09 21:01:26.140557 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:26.140506    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:26.147383 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:26.147335    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:01:26.148391 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:26.148357    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 09 21:01:26.148501 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:26.148429    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 09 21:01:26.344332 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:26.343525    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:26.633334 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:26.633216    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:27.147738 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:27.147694    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 09 21:01:27.147953 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:27.147760    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 09 21:01:28.147399 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:28.147359    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 09 21:01:28.147881 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:28.147427    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 09 21:01:29.147776 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:29.147743    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 09 21:01:29.148222 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:29.148197    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 09 21:01:29.670922 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:29.670891    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:29.732062 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:29.732019    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:01:29.768324 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:29.768273    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:30.147861 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:30.147823    2566 patch_prober.go:28] interesting pod/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused" start-of-body=
Sep 09 21:01:30.148265 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:30.147890    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.58:15021/healthz/ready\": dial tcp 10.132.0.58:15021: connect: connection refused"
Sep 09 21:01:30.326977 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:30.326935    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:30.899380 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:30.899338    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:31.058046 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.058014    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:31.151022 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.150943    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:01:31.152014 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.151995    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:01:31.158375 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.158351    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:31.453988 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.453899    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:31.543707 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.543673    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:31.578649 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.578627    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"]
Sep 09 21:01:31.579083 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.579066    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc94da6-97ab-443c-87d5-439e78e54cbb" containerName="main"
Sep 09 21:01:31.579161 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.579085    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc94da6-97ab-443c-87d5-439e78e54cbb" containerName="main"
Sep 09 21:01:31.579208 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.579166    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc94da6-97ab-443c-87d5-439e78e54cbb" containerName="main"
Sep 09 21:01:31.581247 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.581233    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:31.584045 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.584028    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-0b5d4c\""
Sep 09 21:01:31.584135 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.584077    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-0b5d4c-dockercfg-qgcv8\""
Sep 09 21:01:31.596812 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.596785    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"]
Sep 09 21:01:31.609647 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.609618    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"]
Sep 09 21:01:31.611900 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.611883    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:31.627129 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.627104    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"]
Sep 09 21:01:31.651896 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.651874    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2kb2\" (UniqueName: \"kubernetes.io/projected/cb933b86-35e9-4af2-8ddc-9f6211006896-kube-api-access-t2kb2\") pod \"payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz\" (UID: \"cb933b86-35e9-4af2-8ddc-9f6211006896\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:31.652007 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.651963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb933b86-35e9-4af2-8ddc-9f6211006896-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz\" (UID: \"cb933b86-35e9-4af2-8ddc-9f6211006896\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:31.753423 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.753379    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2kb2\" (UniqueName: \"kubernetes.io/projected/cb933b86-35e9-4af2-8ddc-9f6211006896-kube-api-access-t2kb2\") pod \"payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz\" (UID: \"cb933b86-35e9-4af2-8ddc-9f6211006896\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:31.753622 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.753490    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb933b86-35e9-4af2-8ddc-9f6211006896-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz\" (UID: \"cb933b86-35e9-4af2-8ddc-9f6211006896\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:31.753622 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.753546    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qccb7\" (UniqueName: \"kubernetes.io/projected/224bf361-04d9-4569-9857-aeced3bd1cd4-kube-api-access-qccb7\") pod \"payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8\" (UID: \"224bf361-04d9-4569-9857-aeced3bd1cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:31.753741 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.753610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bf361-04d9-4569-9857-aeced3bd1cd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8\" (UID: \"224bf361-04d9-4569-9857-aeced3bd1cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:31.754241 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.754219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb933b86-35e9-4af2-8ddc-9f6211006896-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz\" (UID: \"cb933b86-35e9-4af2-8ddc-9f6211006896\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:31.761434 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.761407    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2kb2\" (UniqueName: \"kubernetes.io/projected/cb933b86-35e9-4af2-8ddc-9f6211006896-kube-api-access-t2kb2\") pod \"payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz\" (UID: \"cb933b86-35e9-4af2-8ddc-9f6211006896\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:31.855187 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.855139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qccb7\" (UniqueName: \"kubernetes.io/projected/224bf361-04d9-4569-9857-aeced3bd1cd4-kube-api-access-qccb7\") pod \"payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8\" (UID: \"224bf361-04d9-4569-9857-aeced3bd1cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:31.855379 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.855206    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bf361-04d9-4569-9857-aeced3bd1cd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8\" (UID: \"224bf361-04d9-4569-9857-aeced3bd1cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:31.855951 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.855930    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bf361-04d9-4569-9857-aeced3bd1cd4-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8\" (UID: \"224bf361-04d9-4569-9857-aeced3bd1cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:31.864004 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.863980    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qccb7\" (UniqueName: \"kubernetes.io/projected/224bf361-04d9-4569-9857-aeced3bd1cd4-kube-api-access-qccb7\") pod \"payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8\" (UID: \"224bf361-04d9-4569-9857-aeced3bd1cd4\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:31.891042 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.891017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:31.935037 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:31.935012    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:32.165991 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.165965    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"]
Sep 09 21:01:32.168083 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:01:32.168051    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb933b86_35e9_4af2_8ddc_9f6211006896.slice/crio-649a5d6983bccfa5e21520c526f7e8f2f18e6022357b72e90627198818881399 WatchSource:0}: Error finding container 649a5d6983bccfa5e21520c526f7e8f2f18e6022357b72e90627198818881399: Status 404 returned error can't find the container with id 649a5d6983bccfa5e21520c526f7e8f2f18e6022357b72e90627198818881399
Sep 09 21:01:32.217605 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.217583    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"]
Sep 09 21:01:32.219503 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:01:32.219474    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod224bf361_04d9_4569_9857_aeced3bd1cd4.slice/crio-ed0bc8a74996765ea66caebc22f518d876051fb35770959f153c72b20742f9ae WatchSource:0}: Error finding container ed0bc8a74996765ea66caebc22f518d876051fb35770959f153c72b20742f9ae: Status 404 returned error can't find the container with id ed0bc8a74996765ea66caebc22f518d876051fb35770959f153c72b20742f9ae
Sep 09 21:01:32.941876 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.941834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8" event={"ID":"224bf361-04d9-4569-9857-aeced3bd1cd4","Type":"ContainerStarted","Data":"64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc"}
Sep 09 21:01:32.942081 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.941907    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:32.942081 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.941932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8" event={"ID":"224bf361-04d9-4569-9857-aeced3bd1cd4","Type":"ContainerStarted","Data":"ed0bc8a74996765ea66caebc22f518d876051fb35770959f153c72b20742f9ae"}
Sep 09 21:01:32.943457 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.943432    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz" event={"ID":"cb933b86-35e9-4af2-8ddc-9f6211006896","Type":"ContainerStarted","Data":"44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff"}
Sep 09 21:01:32.943588 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.943471    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz" event={"ID":"cb933b86-35e9-4af2-8ddc-9f6211006896","Type":"ContainerStarted","Data":"649a5d6983bccfa5e21520c526f7e8f2f18e6022357b72e90627198818881399"}
Sep 09 21:01:32.943647 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.943607    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:32.965883 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.965827    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8" podStartSLOduration=1.9658081379999999 podStartE2EDuration="1.965808138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:32.962646693 +0000 UTC m=+1273.154484293" watchObservedRunningTime="2026-09-09 21:01:32.965808138 +0000 UTC m=+1273.157645744"
Sep 09 21:01:32.988576 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:32.988520    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz" podStartSLOduration=1.988504894 podStartE2EDuration="1.988504894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:01:32.988498902 +0000 UTC m=+1273.180336519" watchObservedRunningTime="2026-09-09 21:01:32.988504894 +0000 UTC m=+1273.180342493"
Sep 09 21:01:34.470756 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:34.470713    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:35.695500 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:35.695458    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:01:43.949788 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:43.949757    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:01:43.950192 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:43.949834    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:01:46.459413 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:46.459377    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:01:54.708992 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:01:54.708957    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:05.623261 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:05.623217    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:05.696644 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:05.696609    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:12.217148 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:12.217112    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:17.911594 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.911560    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"]
Sep 09 21:02:17.914099 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.914079    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:17.917955 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.917934    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12f2a92f\"/\"e2e-lifecycle-model-12f2a92f-kserve-self-signed-certs\""
Sep 09 21:02:17.918776 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.918757    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12f2a92f\"/\"kube-root-ca.crt\""
Sep 09 21:02:17.918776 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.918763    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-12f2a92f\"/\"default-dockercfg-rkvbs\""
Sep 09 21:02:17.918926 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.918758    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-12f2a92f\"/\"openshift-service-ca.crt\""
Sep 09 21:02:17.934302 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.934275    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"]
Sep 09 21:02:17.958679 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.958645    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:02:17.963700 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.963678    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tmp-dir\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:17.963816 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.963705    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tls-certs\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:17.963816 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.963741    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-home\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:17.963944 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.963924    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-dshm\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:17.964002 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.963969    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82bgb\" (UniqueName: \"kubernetes.io/projected/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-kube-api-access-82bgb\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:17.964059 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:17.964040    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-model-cache\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.065553 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.065526    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-dshm\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.065677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.065559    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82bgb\" (UniqueName: \"kubernetes.io/projected/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-kube-api-access-82bgb\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.065677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.065593    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-model-cache\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.065677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.065623    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tmp-dir\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.065677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.065647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tls-certs\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.065866 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.065806    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-home\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.066020 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.065993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-model-cache\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.066124 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.066047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tmp-dir\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.066124 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.066093    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-home\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.067717 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.067696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-dshm\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.067992 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.067975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tls-certs\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.075355 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.075337    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82bgb\" (UniqueName: \"kubernetes.io/projected/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-kube-api-access-82bgb\") pod \"e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") " pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.225117 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.225035    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:18.505763 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:18.505738    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"]
Sep 09 21:02:18.507232 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:02:18.507206    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb61a6f0f_f6c2_49aa_897d_8d53b4b633e4.slice/crio-764404cf5a947954e960788d67c9a8bca973a0fc72a57790ddb602f90effea87 WatchSource:0}: Error finding container 764404cf5a947954e960788d67c9a8bca973a0fc72a57790ddb602f90effea87: Status 404 returned error can't find the container with id 764404cf5a947954e960788d67c9a8bca973a0fc72a57790ddb602f90effea87
Sep 09 21:02:19.195351 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:19.195299    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z" event={"ID":"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4","Type":"ContainerStarted","Data":"5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454"}
Sep 09 21:02:19.195716 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:19.195358    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z" event={"ID":"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4","Type":"ContainerStarted","Data":"764404cf5a947954e960788d67c9a8bca973a0fc72a57790ddb602f90effea87"}
Sep 09 21:02:19.220466 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:19.220418    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z" podStartSLOduration=2.22040121 podStartE2EDuration="2.22040121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:02:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:02:19.218643899 +0000 UTC m=+1319.410481498" watchObservedRunningTime="2026-09-09 21:02:19.22040121 +0000 UTC m=+1319.412238808"
Sep 09 21:02:22.064653 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:22.064619    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:02:28.225449 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:28.225421    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:28.225948 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:28.225464    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:28.237422 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:28.237400    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:28.254108 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:28.254090    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:02:47.875327 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:47.875235    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:02:54.361163 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:02:54.361112    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:11.168324 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:11.168283    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:03:11.982833 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:11.982802    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"]
Sep 09 21:03:11.985300 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:11.985280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:11.988416 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:11.988395    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d36b91\"/\"unconfigured-w0-d36b91-kserve-self-signed-certs\""
Sep 09 21:03:11.995134 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:11.995112    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"]
Sep 09 21:03:12.018860 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.018820    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:12.055748 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.055725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-dshm\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.055854 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.055765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-model-cache\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.055854 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.055806    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tls-certs\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.056383 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.055856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7t92\" (UniqueName: \"kubernetes.io/projected/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-kube-api-access-q7t92\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.056583 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.056564    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tmp-dir\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.056707 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.056693    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-home\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.157349 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.157294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tmp-dir\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.157349 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.157351    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-home\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.157557 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.157380    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-dshm\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.157557 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.157414    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-model-cache\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.157557 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.157449    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tls-certs\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.157557 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.157525    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7t92\" (UniqueName: \"kubernetes.io/projected/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-kube-api-access-q7t92\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.157760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.157726    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tmp-dir\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.157760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.157749    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-home\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.157876 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.157838    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-model-cache\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.159618 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.159592    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-dshm\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.159876 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.159857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tls-certs\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.165425 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.165401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7t92\" (UniqueName: \"kubernetes.io/projected/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-kube-api-access-q7t92\") pod \"unconfigured-w0-d36b91-kserve-54d4996655-f5zk6\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") " pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.296800 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.296720    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:12.585722 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:12.585698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"]
Sep 09 21:03:12.588045 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:03:12.588017    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod975c6f12_0d20_44c1_8c4d_0b72a6cc5d11.slice/crio-c2278daf4d079ba911a9641198ab2ccb3ccfa9c08b17742fae31a1df08dc9a52 WatchSource:0}: Error finding container c2278daf4d079ba911a9641198ab2ccb3ccfa9c08b17742fae31a1df08dc9a52: Status 404 returned error can't find the container with id c2278daf4d079ba911a9641198ab2ccb3ccfa9c08b17742fae31a1df08dc9a52
Sep 09 21:03:13.483982 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:13.483948    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6" event={"ID":"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11","Type":"ContainerStarted","Data":"0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510"}
Sep 09 21:03:13.484377 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:13.483994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6" event={"ID":"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11","Type":"ContainerStarted","Data":"c2278daf4d079ba911a9641198ab2ccb3ccfa9c08b17742fae31a1df08dc9a52"}
Sep 09 21:03:13.501926 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:13.501873    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6" podStartSLOduration=2.501855814 podStartE2EDuration="2.501855814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:03:13.501500333 +0000 UTC m=+1373.693337935" watchObservedRunningTime="2026-09-09 21:03:13.501855814 +0000 UTC m=+1373.693693413"
Sep 09 21:03:17.832521 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:17.832484    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:17.924217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:17.924181    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:03:22.297394 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:22.297364    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:22.297394 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:22.297403    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:22.309983 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:22.309957    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:22.548582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:22.548505    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:03:39.989390 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:03:39.989356    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:04:03.731025 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:03.730988    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:10.215559 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:10.215518    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:19.362038 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.361956    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:19.418422 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.418392    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"]
Sep 09 21:04:19.418869 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.418832    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv" podUID="58bae4a8-74ba-471f-8001-a109941dcc0e" containerName="payload-processing" containerID="cri-o://246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80" gracePeriod=30
Sep 09 21:04:19.430723 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.430645    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"]
Sep 09 21:04:19.430905 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.430877    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922" podUID="f1be8542-489a-4ed3-9b0e-084a2e3d5e8e" containerName="payload-pre-processing" containerID="cri-o://21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39" gracePeriod=30
Sep 09 21:04:19.471993 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.471947    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:19.550883 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.550835    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:19.636386 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.636249    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:19.687558 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.687534    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:04:19.720522 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.720499    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:04:19.785078 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.785036    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:19.837481 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.837450    2566 generic.go:356] "Generic (PLEG): container finished" podID="f1be8542-489a-4ed3-9b0e-084a2e3d5e8e" containerID="21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39" exitCode=0
Sep 09 21:04:19.837659 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.837495    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"
Sep 09 21:04:19.837659 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.837537    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922" event={"ID":"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e","Type":"ContainerDied","Data":"21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39"}
Sep 09 21:04:19.837659 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.837609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922" event={"ID":"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e","Type":"ContainerDied","Data":"7137d51ae80024ea71851f22c470d203c8273b92f934a5bce3491af99e2ab486"}
Sep 09 21:04:19.837659 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.837640    2566 scope.go:117] "RemoveContainer" containerID="21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39"
Sep 09 21:04:19.838957 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.838935    2566 generic.go:356] "Generic (PLEG): container finished" podID="58bae4a8-74ba-471f-8001-a109941dcc0e" containerID="246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80" exitCode=0
Sep 09 21:04:19.839076 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.838961    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv" event={"ID":"58bae4a8-74ba-471f-8001-a109941dcc0e","Type":"ContainerDied","Data":"246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80"}
Sep 09 21:04:19.839076 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.838980    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"
Sep 09 21:04:19.839076 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.839020    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv" event={"ID":"58bae4a8-74ba-471f-8001-a109941dcc0e","Type":"ContainerDied","Data":"be6344117ad4306cc6ee3982b6d51fae7356d07fac21a277968cb2d0e063c7d6"}
Sep 09 21:04:19.846113 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.846097    2566 scope.go:117] "RemoveContainer" containerID="21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39"
Sep 09 21:04:19.846410 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:04:19.846386    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39\": container with ID starting with 21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39 not found: ID does not exist" containerID="21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39"
Sep 09 21:04:19.846500 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.846418    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39"} err="failed to get container status \"21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39\": rpc error: code = NotFound desc = could not find container \"21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39\": container with ID starting with 21c5f6b178f92ed3d41b1dcb45fb085d4bea8bb461ebace85560246c888c7b39 not found: ID does not exist"
Sep 09 21:04:19.846500 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.846441    2566 scope.go:117] "RemoveContainer" containerID="246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80"
Sep 09 21:04:19.854413 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.854396    2566 scope.go:117] "RemoveContainer" containerID="246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80"
Sep 09 21:04:19.854657 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:04:19.854641    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80\": container with ID starting with 246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80 not found: ID does not exist" containerID="246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80"
Sep 09 21:04:19.854720 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.854664    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80"} err="failed to get container status \"246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80\": rpc error: code = NotFound desc = could not find container \"246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80\": container with ID starting with 246ea15b3d2b9476a67eeb9136b2ef641029bc1117820eb4f5f22cf0edf1cc80 not found: ID does not exist"
Sep 09 21:04:19.871139 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.871112    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pll79\" (UniqueName: \"kubernetes.io/projected/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-kube-api-access-pll79\") pod \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\" (UID: \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\") "
Sep 09 21:04:19.871236 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.871203    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-plugins-config-volume\") pod \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\" (UID: \"f1be8542-489a-4ed3-9b0e-084a2e3d5e8e\") "
Sep 09 21:04:19.871302 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.871273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bae4a8-74ba-471f-8001-a109941dcc0e-plugins-config-volume\") pod \"58bae4a8-74ba-471f-8001-a109941dcc0e\" (UID: \"58bae4a8-74ba-471f-8001-a109941dcc0e\") "
Sep 09 21:04:19.871395 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.871326    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-79zzh\" (UniqueName: \"kubernetes.io/projected/58bae4a8-74ba-471f-8001-a109941dcc0e-kube-api-access-79zzh\") pod \"58bae4a8-74ba-471f-8001-a109941dcc0e\" (UID: \"58bae4a8-74ba-471f-8001-a109941dcc0e\") "
Sep 09 21:04:19.871602 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.871578    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f1be8542-489a-4ed3-9b0e-084a2e3d5e8e" (UID: "f1be8542-489a-4ed3-9b0e-084a2e3d5e8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:19.871602 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.871578    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58bae4a8-74ba-471f-8001-a109941dcc0e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "58bae4a8-74ba-471f-8001-a109941dcc0e" (UID: "58bae4a8-74ba-471f-8001-a109941dcc0e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:04:19.873248 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.873223    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58bae4a8-74ba-471f-8001-a109941dcc0e-kube-api-access-79zzh" (OuterVolumeSpecName: "kube-api-access-79zzh") pod "58bae4a8-74ba-471f-8001-a109941dcc0e" (UID: "58bae4a8-74ba-471f-8001-a109941dcc0e"). InnerVolumeSpecName "kube-api-access-79zzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:19.873433 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.873271    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-kube-api-access-pll79" (OuterVolumeSpecName: "kube-api-access-pll79") pod "f1be8542-489a-4ed3-9b0e-084a2e3d5e8e" (UID: "f1be8542-489a-4ed3-9b0e-084a2e3d5e8e"). InnerVolumeSpecName "kube-api-access-pll79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:19.972195 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.972114    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pll79\" (UniqueName: \"kubernetes.io/projected/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-kube-api-access-pll79\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:19.972195 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.972143    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:19.972195 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.972154    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58bae4a8-74ba-471f-8001-a109941dcc0e-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:19.972195 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:19.972162    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79zzh\" (UniqueName: \"kubernetes.io/projected/58bae4a8-74ba-471f-8001-a109941dcc0e-kube-api-access-79zzh\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:20.161911 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:20.161883    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"]
Sep 09 21:04:20.166371 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:20.166346    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv"]
Sep 09 21:04:20.176364 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:20.176342    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"]
Sep 09 21:04:20.180301 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:20.180278    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922"]
Sep 09 21:04:20.287083 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:20.287053    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58bae4a8-74ba-471f-8001-a109941dcc0e" path="/var/lib/kubelet/pods/58bae4a8-74ba-471f-8001-a109941dcc0e/volumes"
Sep 09 21:04:20.287440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:20.287425    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1be8542-489a-4ed3-9b0e-084a2e3d5e8e" path="/var/lib/kubelet/pods/f1be8542-489a-4ed3-9b0e-084a2e3d5e8e/volumes"
Sep 09 21:04:20.665246 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:20.665148    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-12f2a92f-6496b94986-b8922" podUID="f1be8542-489a-4ed3-9b0e-084a2e3d5e8e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.56:9004: i/o timeout"
Sep 09 21:04:20.665246 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:20.665148    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-12f2a92f-84bff4b99c-l8lsv" podUID="58bae4a8-74ba-471f-8001-a109941dcc0e" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.57:9004: i/o timeout"
Sep 09 21:04:21.526701 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:21.526653    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:22.672647 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:22.672599    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:23.105416 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:23.105360    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:27.646052 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:27.646012    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"]
Sep 09 21:04:27.739812 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:27.739774    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:32.495349 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.495319    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"]
Sep 09 21:04:32.495835 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.495817    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58bae4a8-74ba-471f-8001-a109941dcc0e" containerName="payload-processing"
Sep 09 21:04:32.495890 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.495837    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="58bae4a8-74ba-471f-8001-a109941dcc0e" containerName="payload-processing"
Sep 09 21:04:32.495890 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.495876    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1be8542-489a-4ed3-9b0e-084a2e3d5e8e" containerName="payload-pre-processing"
Sep 09 21:04:32.495890 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.495883    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1be8542-489a-4ed3-9b0e-084a2e3d5e8e" containerName="payload-pre-processing"
Sep 09 21:04:32.496004 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.495961    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="58bae4a8-74ba-471f-8001-a109941dcc0e" containerName="payload-processing"
Sep 09 21:04:32.496004 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.495971    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1be8542-489a-4ed3-9b0e-084a2e3d5e8e" containerName="payload-pre-processing"
Sep 09 21:04:32.498704 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.498690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.501246 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.501223    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"kube-root-ca.crt\""
Sep 09 21:04:32.502121 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.502099    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"openshift-service-ca.crt\""
Sep 09 21:04:32.502227 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.502104    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"default-dockercfg-qrbw2\""
Sep 09 21:04:32.502227 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.502104    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"auth-test-model-0b5d4c-kserve-self-signed-certs\""
Sep 09 21:04:32.509714 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.509694    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"]
Sep 09 21:04:32.581507 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.581476    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcqx4\" (UniqueName: \"kubernetes.io/projected/67a57a69-5d3e-450c-866c-d5b730edb1f0-kube-api-access-rcqx4\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.581672 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.581515    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-dshm\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.581672 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.581565    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a57a69-5d3e-450c-866c-d5b730edb1f0-tls-certs\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.581672 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.581653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-model-cache\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.581803 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.581685    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-tmp-dir\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.581803 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.581728    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-home\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.586560 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.586532    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:32.682596 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.682574    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcqx4\" (UniqueName: \"kubernetes.io/projected/67a57a69-5d3e-450c-866c-d5b730edb1f0-kube-api-access-rcqx4\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.682699 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.682606    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-dshm\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.682699 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.682624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a57a69-5d3e-450c-866c-d5b730edb1f0-tls-certs\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.682699 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.682650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-model-cache\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.682699 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.682671    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-tmp-dir\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.682894 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.682716    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-home\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.683070 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.683046    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-model-cache\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.683174 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.683081    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-home\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.683174 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.683113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-tmp-dir\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.684838 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.684818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-dshm\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.685092 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.685076    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a57a69-5d3e-450c-866c-d5b730edb1f0-tls-certs\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.690396 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.690378    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcqx4\" (UniqueName: \"kubernetes.io/projected/67a57a69-5d3e-450c-866c-d5b730edb1f0-kube-api-access-rcqx4\") pod \"auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:32.808554 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:32.808471    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:33.079207 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:33.079036    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"]
Sep 09 21:04:33.081965 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:04:33.081937    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod67a57a69_5d3e_450c_866c_d5b730edb1f0.slice/crio-9cb4b8a09f11efaf46f998ad6ab5d82ed47f932fb5f6f63659430b5978755f6d WatchSource:0}: Error finding container 9cb4b8a09f11efaf46f998ad6ab5d82ed47f932fb5f6f63659430b5978755f6d: Status 404 returned error can't find the container with id 9cb4b8a09f11efaf46f998ad6ab5d82ed47f932fb5f6f63659430b5978755f6d
Sep 09 21:04:33.922277 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:33.922241    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65" event={"ID":"67a57a69-5d3e-450c-866c-d5b730edb1f0","Type":"ContainerStarted","Data":"ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3"}
Sep 09 21:04:33.922277 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:33.922285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65" event={"ID":"67a57a69-5d3e-450c-866c-d5b730edb1f0","Type":"ContainerStarted","Data":"9cb4b8a09f11efaf46f998ad6ab5d82ed47f932fb5f6f63659430b5978755f6d"}
Sep 09 21:04:33.942173 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:33.942125    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65" podStartSLOduration=1.942112273 podStartE2EDuration="1.942112273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:33.939994485 +0000 UTC m=+1454.131832084" watchObservedRunningTime="2026-09-09 21:04:33.942112273 +0000 UTC m=+1454.133949929"
Sep 09 21:04:34.866395 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:34.866361    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:04:42.670216 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.670182    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z" podUID="b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" containerName="main" containerID="cri-o://5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454" gracePeriod=45
Sep 09 21:04:42.809533 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.809502    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:42.809650 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.809572    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:42.822798 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.822775    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:42.930277 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.930204    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:04:42.972048 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.972013    2566 generic.go:356] "Generic (PLEG): container finished" podID="b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" containerID="5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454" exitCode=0
Sep 09 21:04:42.972194 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.972054    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"
Sep 09 21:04:42.972194 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.972086    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z" event={"ID":"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4","Type":"ContainerDied","Data":"5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454"}
Sep 09 21:04:42.972194 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.972141    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z" event={"ID":"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4","Type":"ContainerDied","Data":"764404cf5a947954e960788d67c9a8bca973a0fc72a57790ddb602f90effea87"}
Sep 09 21:04:42.972194 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.972174    2566 scope.go:117] "RemoveContainer" containerID="5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454"
Sep 09 21:04:42.982358 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.982250    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tmp-dir\") pod \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") "
Sep 09 21:04:42.982874 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.982542    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" (UID: "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:42.983015 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.982992    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tls-certs\") pod \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") "
Sep 09 21:04:42.983119 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.983054    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82bgb\" (UniqueName: \"kubernetes.io/projected/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-kube-api-access-82bgb\") pod \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") "
Sep 09 21:04:42.983181 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.983157    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-dshm\") pod \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") "
Sep 09 21:04:42.983241 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.983193    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-home\") pod \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") "
Sep 09 21:04:42.983241 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.983226    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-model-cache\") pod \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\" (UID: \"b61a6f0f-f6c2-49aa-897d-8d53b4b633e4\") "
Sep 09 21:04:42.983573 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.983547    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:42.983661 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.983577    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-home" (OuterVolumeSpecName: "home") pod "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" (UID: "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:42.983760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.983739    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-model-cache" (OuterVolumeSpecName: "model-cache") pod "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" (UID: "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:42.984001 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.983979    2566 scope.go:117] "RemoveContainer" containerID="5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454"
Sep 09 21:04:42.984412 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:04:42.984300    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454\": container with ID starting with 5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454 not found: ID does not exist" containerID="5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454"
Sep 09 21:04:42.984412 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.984362    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454"} err="failed to get container status \"5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454\": rpc error: code = NotFound desc = could not find container \"5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454\": container with ID starting with 5fa5ec953dc3b97d6b708b8f4bd0015a4fa939e5c3dcfca078a5dce5eece3454 not found: ID does not exist"
Sep 09 21:04:42.985664 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.985630    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-kube-api-access-82bgb" (OuterVolumeSpecName: "kube-api-access-82bgb") pod "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" (UID: "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4"). InnerVolumeSpecName "kube-api-access-82bgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:04:42.985831 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.985812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" (UID: "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:04:42.985831 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.985821    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-dshm" (OuterVolumeSpecName: "dshm") pod "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" (UID: "b61a6f0f-f6c2-49aa-897d-8d53b4b633e4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:04:42.986091 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:42.986075    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:04:43.084737 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:43.084710    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:43.084737 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:43.084739    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82bgb\" (UniqueName: \"kubernetes.io/projected/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-kube-api-access-82bgb\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:43.084904 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:43.084752    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:43.084904 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:43.084764    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:43.084904 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:43.084776    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:04:43.294850 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:43.294823    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"]
Sep 09 21:04:43.298739 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:43.298715    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-12f2a92f/e2e-lifecycle-model-12f2a92f-kserve-5d8cd464ff-vvj5z"]
Sep 09 21:04:44.286542 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:44.286511    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" path="/var/lib/kubelet/pods/b61a6f0f-f6c2-49aa-897d-8d53b4b633e4/volumes"
Sep 09 21:04:47.796934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.796899    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"]
Sep 09 21:04:47.797529 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.797509    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" containerName="main"
Sep 09 21:04:47.797620 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.797531    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" containerName="main"
Sep 09 21:04:47.797675 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.797652    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b61a6f0f-f6c2-49aa-897d-8d53b4b633e4" containerName="main"
Sep 09 21:04:47.801961 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.801936    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:47.804926 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.804904    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-d36b91\"/\"embedding-w0-d36b91-kserve-self-signed-certs\""
Sep 09 21:04:47.811006 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.810984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"]
Sep 09 21:04:47.832930 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.832907    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:04:47.925622 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.925593    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-dshm\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:47.925721 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.925640    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c8g9\" (UniqueName: \"kubernetes.io/projected/eec4de89-5aea-491b-849f-fb7351b9d9ff-kube-api-access-6c8g9\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:47.925721 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.925679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-tmp-dir\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:47.925827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.925725    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-home\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:47.925827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.925762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-model-cache\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:47.925827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:47.925777    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4de89-5aea-491b-849f-fb7351b9d9ff-tls-certs\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.026450 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.026422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c8g9\" (UniqueName: \"kubernetes.io/projected/eec4de89-5aea-491b-849f-fb7351b9d9ff-kube-api-access-6c8g9\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.026637 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.026465    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-tmp-dir\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.026637 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.026515    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-home\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.026637 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.026533    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-model-cache\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.026637 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.026548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4de89-5aea-491b-849f-fb7351b9d9ff-tls-certs\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.026637 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.026620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-dshm\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.026980 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.026959    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-home\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.027036 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.026992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-tmp-dir\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.027096 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.027056    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-model-cache\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.028713 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.028696    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-dshm\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.029048 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.029031    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4de89-5aea-491b-849f-fb7351b9d9ff-tls-certs\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.035653 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.035635    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c8g9\" (UniqueName: \"kubernetes.io/projected/eec4de89-5aea-491b-849f-fb7351b9d9ff-kube-api-access-6c8g9\") pod \"embedding-w0-d36b91-kserve-868c865f7-pdkxq\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") " pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.111351 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.111293    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:48.392236 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:48.392213    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"]
Sep 09 21:04:48.395016 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:04:48.394989    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podeec4de89_5aea_491b_849f_fb7351b9d9ff.slice/crio-a7a5d3ec4df9084e35a57747309692c9ed85bbb00fcb6b52ea2f83b44c5c1f3b WatchSource:0}: Error finding container a7a5d3ec4df9084e35a57747309692c9ed85bbb00fcb6b52ea2f83b44c5c1f3b: Status 404 returned error can't find the container with id a7a5d3ec4df9084e35a57747309692c9ed85bbb00fcb6b52ea2f83b44c5c1f3b
Sep 09 21:04:49.006678 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:49.006640    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq" event={"ID":"eec4de89-5aea-491b-849f-fb7351b9d9ff","Type":"ContainerStarted","Data":"b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455"}
Sep 09 21:04:49.007049 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:49.006688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq" event={"ID":"eec4de89-5aea-491b-849f-fb7351b9d9ff","Type":"ContainerStarted","Data":"a7a5d3ec4df9084e35a57747309692c9ed85bbb00fcb6b52ea2f83b44c5c1f3b"}
Sep 09 21:04:49.026653 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:49.026604    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq" podStartSLOduration=2.026590451 podStartE2EDuration="2.026590451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:04:49.023616067 +0000 UTC m=+1469.215453666" watchObservedRunningTime="2026-09-09 21:04:49.026590451 +0000 UTC m=+1469.218428050"
Sep 09 21:04:58.111981 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:58.111940    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:58.112423 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:58.111996    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:58.125293 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:58.125268    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:04:59.070011 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:04:59.069983    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:05:03.111262 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:03.111227    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:04.570564 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:04.570527    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:04.983347 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:04.983279    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:05.665743 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:05.665713    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:05:10.816397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:10.816362    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:25.637106 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:25.637069    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:25.737500 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:25.737460    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:25.903351 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:25.903252    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"]
Sep 09 21:05:25.903516 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:25.903474    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy" containerID="cri-o://c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f" gracePeriod=30
Sep 09 21:05:25.934327 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:25.934282    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:27.096153 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:27.096109    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:27.186344 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:27.186286    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:27.284547 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:27.284509    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:27.853338 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:27.853286    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:28.010117 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:28.010080    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:29.275283 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:29.275246    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:31.133390 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.133354    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:31.164300 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.164272    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:05:31.209559 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209469    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-certs\") pod \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") "
Sep 09 21:05:31.209559 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209525    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-credential-socket\") pod \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") "
Sep 09 21:05:31.209790 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-podinfo\") pod \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") "
Sep 09 21:05:31.209790 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209596    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-socket\") pod \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") "
Sep 09 21:05:31.209790 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209653    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-token\") pod \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") "
Sep 09 21:05:31.209790 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209685    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-envoy\") pod \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") "
Sep 09 21:05:31.209790 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209719    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7b5e1ec7-a190-4397-b8ff-cc4ec236340e" (UID: "7b5e1ec7-a190-4397-b8ff-cc4ec236340e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.209790 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209733    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-data\") pod \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") "
Sep 09 21:05:31.209790 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209785    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istiod-ca-cert\") pod \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") "
Sep 09 21:05:31.210217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209820    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbzdq\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-kube-api-access-kbzdq\") pod \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\" (UID: \"7b5e1ec7-a190-4397-b8ff-cc4ec236340e\") "
Sep 09 21:05:31.210217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209881    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7b5e1ec7-a190-4397-b8ff-cc4ec236340e" (UID: "7b5e1ec7-a190-4397-b8ff-cc4ec236340e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.210217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209938    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-data" (OuterVolumeSpecName: "istio-data") pod "7b5e1ec7-a190-4397-b8ff-cc4ec236340e" (UID: "7b5e1ec7-a190-4397-b8ff-cc4ec236340e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.210217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.209959    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7b5e1ec7-a190-4397-b8ff-cc4ec236340e" (UID: "7b5e1ec7-a190-4397-b8ff-cc4ec236340e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.210217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.210133    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7b5e1ec7-a190-4397-b8ff-cc4ec236340e" (UID: "7b5e1ec7-a190-4397-b8ff-cc4ec236340e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:05:31.210516 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.210259    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-data\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.210516 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.210277    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istiod-ca-cert\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.210516 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.210291    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.210516 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.210303    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-credential-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.210516 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.210335    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-workload-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.211971 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.211926    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7b5e1ec7-a190-4397-b8ff-cc4ec236340e" (UID: "7b5e1ec7-a190-4397-b8ff-cc4ec236340e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:05:31.212079 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.212037    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-token" (OuterVolumeSpecName: "istio-token") pod "7b5e1ec7-a190-4397-b8ff-cc4ec236340e" (UID: "7b5e1ec7-a190-4397-b8ff-cc4ec236340e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:31.212263 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.212240    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-kube-api-access-kbzdq" (OuterVolumeSpecName: "kube-api-access-kbzdq") pod "7b5e1ec7-a190-4397-b8ff-cc4ec236340e" (UID: "7b5e1ec7-a190-4397-b8ff-cc4ec236340e"). InnerVolumeSpecName "kube-api-access-kbzdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:05:31.212263 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.212251    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7b5e1ec7-a190-4397-b8ff-cc4ec236340e" (UID: "7b5e1ec7-a190-4397-b8ff-cc4ec236340e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:05:31.230616 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.230584    2566 generic.go:356] "Generic (PLEG): container finished" podID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerID="c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f" exitCode=0
Sep 09 21:05:31.230737 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.230627    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"
Sep 09 21:05:31.230737 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.230638    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" event={"ID":"7b5e1ec7-a190-4397-b8ff-cc4ec236340e","Type":"ContainerDied","Data":"c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f"}
Sep 09 21:05:31.230737 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.230693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh" event={"ID":"7b5e1ec7-a190-4397-b8ff-cc4ec236340e","Type":"ContainerDied","Data":"6e1f0ed37ef540d878c94848a08cf7ca43d52df69575a2c3c41c86a7aff56d10"}
Sep 09 21:05:31.230737 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.230723    2566 scope.go:117] "RemoveContainer" containerID="c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f"
Sep 09 21:05:31.240213 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.240195    2566 scope.go:117] "RemoveContainer" containerID="6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21"
Sep 09 21:05:31.248253 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.248218    2566 scope.go:117] "RemoveContainer" containerID="c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f"
Sep 09 21:05:31.248534 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:05:31.248516    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f\": container with ID starting with c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f not found: ID does not exist" containerID="c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f"
Sep 09 21:05:31.248600 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.248542    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f"} err="failed to get container status \"c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f\": rpc error: code = NotFound desc = could not find container \"c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f\": container with ID starting with c1b953be2e2b0f2a09e711204879fd849d1a82f8ba4941951b15d8bc3ee30f7f not found: ID does not exist"
Sep 09 21:05:31.248600 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.248557    2566 scope.go:117] "RemoveContainer" containerID="6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21"
Sep 09 21:05:31.248801 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:05:31.248788    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21\": container with ID starting with 6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21 not found: ID does not exist" containerID="6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21"
Sep 09 21:05:31.248846 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.248803    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21"} err="failed to get container status \"6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21\": rpc error: code = NotFound desc = could not find container \"6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21\": container with ID starting with 6b11a5871a31a83ca52791fef827875f427a8af6daaa43224a057a8847245e21 not found: ID does not exist"
Sep 09 21:05:31.253475 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.253452    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"]
Sep 09 21:05:31.257144 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.257117    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-12f2a92f-openshift-default-6b8f9ccb56-nmsdh"]
Sep 09 21:05:31.311471 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.311441    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-token\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.311471 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.311464    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-envoy\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.311471 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.311476    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbzdq\" (UniqueName: \"kubernetes.io/projected/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-kube-api-access-kbzdq\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:05:31.311757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:31.311484    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b5e1ec7-a190-4397-b8ff-cc4ec236340e-istio-podinfo\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:05:32.286803 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:32.286765    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" path="/var/lib/kubelet/pods/7b5e1ec7-a190-4397-b8ff-cc4ec236340e/volumes"
Sep 09 21:05:36.593608 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:36.593552    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:36.695183 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:36.695139    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:41.940126 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:41.940092    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:42.054264 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:42.054228    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:42.158673 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:42.158638    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:44.587385 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.587283    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"]
Sep 09 21:05:44.587988 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.587966    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy"
Sep 09 21:05:44.588055 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.587994    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy"
Sep 09 21:05:44.588055 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.588037    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy"
Sep 09 21:05:44.588055 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.588046    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy"
Sep 09 21:05:44.588162 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.588148    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy"
Sep 09 21:05:44.588197 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.588163    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b5e1ec7-a190-4397-b8ff-cc4ec236340e" containerName="istio-proxy"
Sep 09 21:05:44.593474 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.593451    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.596660 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.596623    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-47e41c\"/\"e2e-conflict-a-98510255-kserve-self-signed-certs\""
Sep 09 21:05:44.601645 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.601621    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"]
Sep 09 21:05:44.623871 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.623828    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:44.736601 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.736561    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-model-cache\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.736771 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.736658    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-tmp-dir\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.736771 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.736733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mnnlc\" (UniqueName: \"kubernetes.io/projected/99d425c6-5fde-408e-a431-8fdc782c9164-kube-api-access-mnnlc\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.736771 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.736756    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-dshm\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.736888 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.736780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d425c6-5fde-408e-a431-8fdc782c9164-tls-certs\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.736888 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.736862    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-home\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.837642 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.837557    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-model-cache\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.837808 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.837646    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-tmp-dir\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.837808 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.837727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mnnlc\" (UniqueName: \"kubernetes.io/projected/99d425c6-5fde-408e-a431-8fdc782c9164-kube-api-access-mnnlc\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.837808 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.837761    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-dshm\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.837808 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.837796    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d425c6-5fde-408e-a431-8fdc782c9164-tls-certs\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.838035 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.837847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-home\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.838035 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.837912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-model-cache\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.838035 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.837960    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-tmp-dir\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.838246 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.838225    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-home\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.840177 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.840150    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-dshm\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.840353 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.840334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d425c6-5fde-408e-a431-8fdc782c9164-tls-certs\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.846079 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.846054    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mnnlc\" (UniqueName: \"kubernetes.io/projected/99d425c6-5fde-408e-a431-8fdc782c9164-kube-api-access-mnnlc\") pod \"e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:44.904459 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:44.904418    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:45.177676 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:45.177652    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"]
Sep 09 21:05:45.179726 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:05:45.179699    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99d425c6_5fde_408e_a431_8fdc782c9164.slice/crio-9f83fbb1ff15df25bd6ece4243717cf68845f949e6837c99cf14f30d185f7e36 WatchSource:0}: Error finding container 9f83fbb1ff15df25bd6ece4243717cf68845f949e6837c99cf14f30d185f7e36: Status 404 returned error can't find the container with id 9f83fbb1ff15df25bd6ece4243717cf68845f949e6837c99cf14f30d185f7e36
Sep 09 21:05:45.302914 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:45.302872    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc" event={"ID":"99d425c6-5fde-408e-a431-8fdc782c9164","Type":"ContainerStarted","Data":"c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f"}
Sep 09 21:05:45.303106 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:45.302930    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc" event={"ID":"99d425c6-5fde-408e-a431-8fdc782c9164","Type":"ContainerStarted","Data":"9f83fbb1ff15df25bd6ece4243717cf68845f949e6837c99cf14f30d185f7e36"}
Sep 09 21:05:45.322069 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:45.322011    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc" podStartSLOduration=1.3219940110000001 podStartE2EDuration="1.321994011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:05:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:05:45.320080559 +0000 UTC m=+1525.511918159" watchObservedRunningTime="2026-09-09 21:05:45.321994011 +0000 UTC m=+1525.513831609"
Sep 09 21:05:52.654827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:52.654789    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:52.984939 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:52.984901    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:54.905124 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:54.905094    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:54.905632 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:54.905164    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:54.917867 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:54.917850    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:55.364736 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:55.364705    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:05:55.560868 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:55.560834    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:57.462137 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:57.462100    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:05:58.045684 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:05:58.045650    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:04.783464 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.783430    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"]
Sep 09 21:06:04.786800 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.786784    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.793619 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.793595    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-47e41c\"/\"e2e-conflict-b-98510255-kserve-self-signed-certs\""
Sep 09 21:06:04.801507 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.801485    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"]
Sep 09 21:06:04.837397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.833963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tls-certs\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.837397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.834032    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-model-cache\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.837397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.834070    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-dshm\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.837397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.834155    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-home\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.837397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.834201    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sgb7\" (UniqueName: \"kubernetes.io/projected/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-kube-api-access-8sgb7\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.837397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.834254    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tmp-dir\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.839979 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.839826    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:04.935752 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.935721    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tls-certs\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.935752 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.935764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-model-cache\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.935982 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.935785    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-dshm\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.935982 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.935830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-home\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.935982 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.935856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sgb7\" (UniqueName: \"kubernetes.io/projected/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-kube-api-access-8sgb7\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.935982 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.935889    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tmp-dir\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.936214 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.936187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-model-cache\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.936287 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.936233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-home\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.936375 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.936354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tmp-dir\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.938040 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.938022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-dshm\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.938357 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.938334    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tls-certs\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:04.945510 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:04.945486    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sgb7\" (UniqueName: \"kubernetes.io/projected/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-kube-api-access-8sgb7\") pod \"e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") " pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:05.098007 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:05.097926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:05.376640 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:05.376611    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"]
Sep 09 21:06:05.378232 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:06:05.378205    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d5d96a6_d3ee_41fd_ac2c_1cbbae078b15.slice/crio-28745deb9673ca5905c4da064b806fe03019af1a3b8f656245b61eae78307a76 WatchSource:0}: Error finding container 28745deb9673ca5905c4da064b806fe03019af1a3b8f656245b61eae78307a76: Status 404 returned error can't find the container with id 28745deb9673ca5905c4da064b806fe03019af1a3b8f656245b61eae78307a76
Sep 09 21:06:05.403527 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:05.403496    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4" event={"ID":"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15","Type":"ContainerStarted","Data":"28745deb9673ca5905c4da064b806fe03019af1a3b8f656245b61eae78307a76"}
Sep 09 21:06:06.412211 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:06.412170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4" event={"ID":"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15","Type":"ContainerStarted","Data":"5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2"}
Sep 09 21:06:06.430025 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:06.429972    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4" podStartSLOduration=2.429954971 podStartE2EDuration="2.429954971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:06.428896568 +0000 UTC m=+1546.620734158" watchObservedRunningTime="2026-09-09 21:06:06.429954971 +0000 UTC m=+1546.621792570"
Sep 09 21:06:13.761714 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:13.761664    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:15.098508 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:15.098460    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:15.098910 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:15.098811    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:15.111239 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:15.111211    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:15.474290 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:15.474207    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:16.366408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:16.366364    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:20.299543 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:20.299515    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:20.401596 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:20.401552    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:20.645821 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:20.645734    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:20.981787 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:20.981753    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:21.091539 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:21.091505    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:23.303710 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:23.303678    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"]
Sep 09 21:06:23.333477 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:23.333449    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:23.965458 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:23.965419    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:24.469798 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:24.469765    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:24.573874 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:24.573845    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:24.978229 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:24.978197    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"]
Sep 09 21:06:25.003934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:25.003904    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:25.728592 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:25.728561    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:25.866707 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:25.866674    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:25.956141 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:25.956097    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:26.274817 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:26.274749    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:26.433733 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:26.433696    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:35.403881 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:35.403841    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:06:38.323767 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.323735    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4" podUID="2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" containerName="main" containerID="cri-o://5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2" gracePeriod=45
Sep 09 21:06:38.583178 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.583105    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:38.594713 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.594685    2566 generic.go:356] "Generic (PLEG): container finished" podID="2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" containerID="5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2" exitCode=0
Sep 09 21:06:38.594855 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.594724    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"
Sep 09 21:06:38.594855 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.594731    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4" event={"ID":"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15","Type":"ContainerDied","Data":"5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2"}
Sep 09 21:06:38.594855 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.594847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4" event={"ID":"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15","Type":"ContainerDied","Data":"28745deb9673ca5905c4da064b806fe03019af1a3b8f656245b61eae78307a76"}
Sep 09 21:06:38.594991 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.594868    2566 scope.go:117] "RemoveContainer" containerID="5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2"
Sep 09 21:06:38.604716 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.604695    2566 scope.go:117] "RemoveContainer" containerID="5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2"
Sep 09 21:06:38.605064 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:06:38.605035    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2\": container with ID starting with 5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2 not found: ID does not exist" containerID="5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2"
Sep 09 21:06:38.605171 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.605076    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2"} err="failed to get container status \"5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2\": rpc error: code = NotFound desc = could not find container \"5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2\": container with ID starting with 5d86386d33f743ef9edd5545b136f00fd05ebd172c6eac7320d1ffc405673ea2 not found: ID does not exist"
Sep 09 21:06:38.757934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.757893    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tls-certs\") pod \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") "
Sep 09 21:06:38.758108 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.757979    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-home\") pod \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") "
Sep 09 21:06:38.758108 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.758001    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-model-cache\") pod \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") "
Sep 09 21:06:38.758108 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.758019    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-dshm\") pod \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") "
Sep 09 21:06:38.758282 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.758112    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tmp-dir\") pod \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") "
Sep 09 21:06:38.758282 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.758141    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sgb7\" (UniqueName: \"kubernetes.io/projected/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-kube-api-access-8sgb7\") pod \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\" (UID: \"2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15\") "
Sep 09 21:06:38.758408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.758272    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-home" (OuterVolumeSpecName: "home") pod "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" (UID: "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:38.758408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.758285    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-model-cache" (OuterVolumeSpecName: "model-cache") pod "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" (UID: "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:38.758562 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.758540    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:38.758562 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.758548    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" (UID: "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:38.758562 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.758565    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:38.760258 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.760236    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-dshm" (OuterVolumeSpecName: "dshm") pod "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" (UID: "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:38.760589 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.760560    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" (UID: "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:38.760589 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.760579    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-kube-api-access-8sgb7" (OuterVolumeSpecName: "kube-api-access-8sgb7") pod "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" (UID: "2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15"). InnerVolumeSpecName "kube-api-access-8sgb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:38.861702 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.860699    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:38.861702 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.860731    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:38.861702 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.860744    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:38.861702 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.860758    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sgb7\" (UniqueName: \"kubernetes.io/projected/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15-kube-api-access-8sgb7\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:38.920045 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.920017    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"]
Sep 09 21:06:38.926957 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:38.926928    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-b-98510255-kserve-846685f6c4-2f5s4"]
Sep 09 21:06:39.997733 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:39.997686    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc" podUID="99d425c6-5fde-408e-a431-8fdc782c9164" containerName="main" containerID="cri-o://c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f" gracePeriod=45
Sep 09 21:06:40.254103 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.254049    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:06:40.275713 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.275684    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-dshm\") pod \"99d425c6-5fde-408e-a431-8fdc782c9164\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") "
Sep 09 21:06:40.275839 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.275724    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mnnlc\" (UniqueName: \"kubernetes.io/projected/99d425c6-5fde-408e-a431-8fdc782c9164-kube-api-access-mnnlc\") pod \"99d425c6-5fde-408e-a431-8fdc782c9164\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") "
Sep 09 21:06:40.275839 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.275756    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d425c6-5fde-408e-a431-8fdc782c9164-tls-certs\") pod \"99d425c6-5fde-408e-a431-8fdc782c9164\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") "
Sep 09 21:06:40.275839 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.275784    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-tmp-dir\") pod \"99d425c6-5fde-408e-a431-8fdc782c9164\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") "
Sep 09 21:06:40.275839 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.275801    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-model-cache\") pod \"99d425c6-5fde-408e-a431-8fdc782c9164\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") "
Sep 09 21:06:40.276057 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.275843    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-home\") pod \"99d425c6-5fde-408e-a431-8fdc782c9164\" (UID: \"99d425c6-5fde-408e-a431-8fdc782c9164\") "
Sep 09 21:06:40.276149 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.276125    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-model-cache" (OuterVolumeSpecName: "model-cache") pod "99d425c6-5fde-408e-a431-8fdc782c9164" (UID: "99d425c6-5fde-408e-a431-8fdc782c9164"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:40.276213 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.276141    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "99d425c6-5fde-408e-a431-8fdc782c9164" (UID: "99d425c6-5fde-408e-a431-8fdc782c9164"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:40.276475 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.276404    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-home" (OuterVolumeSpecName: "home") pod "99d425c6-5fde-408e-a431-8fdc782c9164" (UID: "99d425c6-5fde-408e-a431-8fdc782c9164"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:40.278473 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.278441    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-dshm" (OuterVolumeSpecName: "dshm") pod "99d425c6-5fde-408e-a431-8fdc782c9164" (UID: "99d425c6-5fde-408e-a431-8fdc782c9164"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:06:40.278473 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.278463    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d425c6-5fde-408e-a431-8fdc782c9164-kube-api-access-mnnlc" (OuterVolumeSpecName: "kube-api-access-mnnlc") pod "99d425c6-5fde-408e-a431-8fdc782c9164" (UID: "99d425c6-5fde-408e-a431-8fdc782c9164"). InnerVolumeSpecName "kube-api-access-mnnlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:06:40.278621 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.278482    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d425c6-5fde-408e-a431-8fdc782c9164-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "99d425c6-5fde-408e-a431-8fdc782c9164" (UID: "99d425c6-5fde-408e-a431-8fdc782c9164"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:06:40.289646 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.289622    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" path="/var/lib/kubelet/pods/2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15/volumes"
Sep 09 21:06:40.376607 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.376576    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d425c6-5fde-408e-a431-8fdc782c9164-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:40.376607 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.376600    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:40.376607 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.376609    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:40.376813 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.376618    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:40.376813 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.376625    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d425c6-5fde-408e-a431-8fdc782c9164-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:40.376813 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.376633    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mnnlc\" (UniqueName: \"kubernetes.io/projected/99d425c6-5fde-408e-a431-8fdc782c9164-kube-api-access-mnnlc\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:06:40.606732 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.606654    2566 generic.go:356] "Generic (PLEG): container finished" podID="99d425c6-5fde-408e-a431-8fdc782c9164" containerID="c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f" exitCode=0
Sep 09 21:06:40.606732 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.606701    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"
Sep 09 21:06:40.606929 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.606739    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc" event={"ID":"99d425c6-5fde-408e-a431-8fdc782c9164","Type":"ContainerDied","Data":"c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f"}
Sep 09 21:06:40.606929 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.606784    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc" event={"ID":"99d425c6-5fde-408e-a431-8fdc782c9164","Type":"ContainerDied","Data":"9f83fbb1ff15df25bd6ece4243717cf68845f949e6837c99cf14f30d185f7e36"}
Sep 09 21:06:40.606929 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.606806    2566 scope.go:117] "RemoveContainer" containerID="c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f"
Sep 09 21:06:40.616353 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.616327    2566 scope.go:117] "RemoveContainer" containerID="c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f"
Sep 09 21:06:40.616615 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:06:40.616596    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f\": container with ID starting with c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f not found: ID does not exist" containerID="c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f"
Sep 09 21:06:40.616675 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.616626    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f"} err="failed to get container status \"c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f\": rpc error: code = NotFound desc = could not find container \"c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f\": container with ID starting with c5bac01b33c4e4d056ebf42d6f7f58d1b106a670a87ac51ca1a832ae1f99de7f not found: ID does not exist"
Sep 09 21:06:40.630673 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.630642    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"]
Sep 09 21:06:40.642058 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:40.642033    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/e2e-conflict-a-98510255-kserve-57c68cdc6-9w8hc"]
Sep 09 21:06:42.286411 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:42.286376    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99d425c6-5fde-408e-a431-8fdc782c9164" path="/var/lib/kubelet/pods/99d425c6-5fde-408e-a431-8fdc782c9164/volumes"
Sep 09 21:06:45.180876 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:45.180843    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:45.451469 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:45.451391    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:48.026811 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:48.026772    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:49.357777 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:49.357739    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:49.809811 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:49.809776    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:50.074924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.069932    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"]
Sep 09 21:06:50.074924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.070926    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99d425c6-5fde-408e-a431-8fdc782c9164" containerName="main"
Sep 09 21:06:50.074924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.070947    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="99d425c6-5fde-408e-a431-8fdc782c9164" containerName="main"
Sep 09 21:06:50.074924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.070978    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" containerName="main"
Sep 09 21:06:50.074924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.070986    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" containerName="main"
Sep 09 21:06:50.074924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.071242    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d5d96a6-d3ee-41fd-ac2c-1cbbae078b15" containerName="main"
Sep 09 21:06:50.074924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.071262    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="99d425c6-5fde-408e-a431-8fdc782c9164" containerName="main"
Sep 09 21:06:50.081989 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.081965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.085076 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.085055    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-0622e2-openshift-default-dockercfg-wzkjx\""
Sep 09 21:06:50.088919 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.088894    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"]
Sep 09 21:06:50.109354 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.109320    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:50.174097 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.174057    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-istio-token\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.174247 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.174113    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-data\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.174247 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.174157    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvgkt\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-kube-api-access-nvgkt\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.174247 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.174190    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-certs\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.174247 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.174220    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e45df-f867-40f3-819e-db13c8b74726-istiod-ca-cert\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.174247 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.174239    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e45df-f867-40f3-819e-db13c8b74726-istio-podinfo\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.174453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.174260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-socket\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.174453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.174340    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-envoy\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.174453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.174379    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-credential-socket\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.234436 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.234400    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"]
Sep 09 21:06:50.242677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.242647    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"]
Sep 09 21:06:50.258972 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.258941    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"]
Sep 09 21:06:50.271180 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.271146    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"]
Sep 09 21:06:50.275974 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.275944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e45df-f867-40f3-819e-db13c8b74726-istiod-ca-cert\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.276144 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.275985    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e45df-f867-40f3-819e-db13c8b74726-istio-podinfo\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.276144 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-socket\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.276144 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-envoy\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.276144 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-credential-socket\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.276144 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-istio-token\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.276437 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276287    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-data\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.276437 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvgkt\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-kube-api-access-nvgkt\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.276437 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276420    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-certs\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.277034 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276774    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e45df-f867-40f3-819e-db13c8b74726-istiod-ca-cert\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.277034 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276861    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-socket\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.277034 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.276988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-credential-socket\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.277232 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.277170    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-data\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.277232 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.277222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-certs\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.279562 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.279536    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-envoy\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.279721 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.279702    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e45df-f867-40f3-819e-db13c8b74726-istio-podinfo\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.294381 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.294320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-istio-token\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.299109 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.299082    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvgkt\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-kube-api-access-nvgkt\") pod \"e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") " pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.303650 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.303620    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:50.374174 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.374079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:50.393425 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.393395    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:50.677585 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:06:50.677546    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d3e45df_f867_40f3_819e_db13c8b74726.slice/crio-ce3534e15b489de313639c0483afbba2d705eafcb57501b1baa18db81798e813 WatchSource:0}: Error finding container ce3534e15b489de313639c0483afbba2d705eafcb57501b1baa18db81798e813: Status 404 returned error can't find the container with id ce3534e15b489de313639c0483afbba2d705eafcb57501b1baa18db81798e813
Sep 09 21:06:50.678586 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.678562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"]
Sep 09 21:06:50.680104 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.680073    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:06:50.680222 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.680149    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:06:50.680222 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:50.680188    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:06:51.091393 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:51.091360    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:51.667874 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:51.667831    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" event={"ID":"0d3e45df-f867-40f3-819e-db13c8b74726","Type":"ContainerStarted","Data":"069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751"}
Sep 09 21:06:51.668362 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:51.667892    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" event={"ID":"0d3e45df-f867-40f3-819e-db13c8b74726","Type":"ContainerStarted","Data":"ce3534e15b489de313639c0483afbba2d705eafcb57501b1baa18db81798e813"}
Sep 09 21:06:51.695074 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:51.695011    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" podStartSLOduration=1.6949942390000001 podStartE2EDuration="1.694994239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:51.692576912 +0000 UTC m=+1591.884414511" watchObservedRunningTime="2026-09-09 21:06:51.694994239 +0000 UTC m=+1591.886831838"
Sep 09 21:06:52.393953 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:52.393914    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:52.399580 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:52.399549    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:52.673151 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:52.673059    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:52.674077 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:52.674057    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:06:53.633435 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:53.633399    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:53.733562 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:53.733521    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:53.883843 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:53.883738    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:54.877137 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:54.877102    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:54.979063 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:54.979012    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:55.104077 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.104039    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:55.627349 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.627294    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:55.723326 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.723276    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:06:55.724890 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.724860    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"]
Sep 09 21:06:55.729020 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.729002    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:06:55.732095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.732075    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-0622e2\""
Sep 09 21:06:55.745023 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.744999    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"]
Sep 09 21:06:55.833261 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.833223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6879aeb-520d-45d2-a01c-1eff17166403-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn\" (UID: \"d6879aeb-520d-45d2-a01c-1eff17166403\") " pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:06:55.833448 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.833278    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nkv\" (UniqueName: \"kubernetes.io/projected/d6879aeb-520d-45d2-a01c-1eff17166403-kube-api-access-j9nkv\") pod \"payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn\" (UID: \"d6879aeb-520d-45d2-a01c-1eff17166403\") " pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:06:55.934958 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.934870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6879aeb-520d-45d2-a01c-1eff17166403-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn\" (UID: \"d6879aeb-520d-45d2-a01c-1eff17166403\") " pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:06:55.934958 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.934934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nkv\" (UniqueName: \"kubernetes.io/projected/d6879aeb-520d-45d2-a01c-1eff17166403-kube-api-access-j9nkv\") pod \"payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn\" (UID: \"d6879aeb-520d-45d2-a01c-1eff17166403\") " pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:06:55.935616 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.935597    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6879aeb-520d-45d2-a01c-1eff17166403-plugins-config-volume\") pod \"payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn\" (UID: \"d6879aeb-520d-45d2-a01c-1eff17166403\") " pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:06:55.947549 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:55.947512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nkv\" (UniqueName: \"kubernetes.io/projected/d6879aeb-520d-45d2-a01c-1eff17166403-kube-api-access-j9nkv\") pod \"payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn\" (UID: \"d6879aeb-520d-45d2-a01c-1eff17166403\") " pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:06:56.039513 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:56.039468    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:06:56.313070 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:56.313043    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"]
Sep 09 21:06:56.315347 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:06:56.315322    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6879aeb_520d_45d2_a01c_1eff17166403.slice/crio-b80445788b49c38bf3285c80bb665b42d7bd006df8c3106ae31ce55edb2dea76 WatchSource:0}: Error finding container b80445788b49c38bf3285c80bb665b42d7bd006df8c3106ae31ce55edb2dea76: Status 404 returned error can't find the container with id b80445788b49c38bf3285c80bb665b42d7bd006df8c3106ae31ce55edb2dea76
Sep 09 21:06:56.695275 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:56.695179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn" event={"ID":"d6879aeb-520d-45d2-a01c-1eff17166403","Type":"ContainerStarted","Data":"542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283"}
Sep 09 21:06:56.695275 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:56.695230    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn" event={"ID":"d6879aeb-520d-45d2-a01c-1eff17166403","Type":"ContainerStarted","Data":"b80445788b49c38bf3285c80bb665b42d7bd006df8c3106ae31ce55edb2dea76"}
Sep 09 21:06:56.695530 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:56.695436    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:06:56.721247 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:06:56.721182    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn" podStartSLOduration=1.7211610670000002 podStartE2EDuration="1.721161067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:06:56.716619988 +0000 UTC m=+1596.908457586" watchObservedRunningTime="2026-09-09 21:06:56.721161067 +0000 UTC m=+1596.912998668"
Sep 09 21:07:01.062986 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.062947    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:01.063731 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.063708    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z"]
Sep 09 21:07:01.072898 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.072871    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-d36b91-29816460-86d8z"]
Sep 09 21:07:01.144049 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.144013    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"]
Sep 09 21:07:01.144260 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.144232    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf" podUID="4eb83949-b8fa-46d5-8fec-3d1a8b823b28" containerName="payload-processing" containerID="cri-o://0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d" gracePeriod=30
Sep 09 21:07:01.274442 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.274405    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:01.386874 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.386845    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:01.409038 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.409010    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 21:07:01.484786 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.484752    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:01.491726 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.491699    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b87m6\" (UniqueName: \"kubernetes.io/projected/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-kube-api-access-b87m6\") pod \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\" (UID: \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\") "
Sep 09 21:07:01.491862 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.491775    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-plugins-config-volume\") pod \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\" (UID: \"4eb83949-b8fa-46d5-8fec-3d1a8b823b28\") "
Sep 09 21:07:01.492189 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.492164    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4eb83949-b8fa-46d5-8fec-3d1a8b823b28" (UID: "4eb83949-b8fa-46d5-8fec-3d1a8b823b28"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:07:01.493965 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.493942    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-kube-api-access-b87m6" (OuterVolumeSpecName: "kube-api-access-b87m6") pod "4eb83949-b8fa-46d5-8fec-3d1a8b823b28" (UID: "4eb83949-b8fa-46d5-8fec-3d1a8b823b28"). InnerVolumeSpecName "kube-api-access-b87m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:01.562635 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.562597    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:01.592801 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.592723    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b87m6\" (UniqueName: \"kubernetes.io/projected/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-kube-api-access-b87m6\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:01.592801 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.592749    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4eb83949-b8fa-46d5-8fec-3d1a8b823b28-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:01.721571 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.721534    2566 generic.go:356] "Generic (PLEG): container finished" podID="4eb83949-b8fa-46d5-8fec-3d1a8b823b28" containerID="0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d" exitCode=0
Sep 09 21:07:01.721743 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.721594    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"
Sep 09 21:07:01.721743 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.721627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf" event={"ID":"4eb83949-b8fa-46d5-8fec-3d1a8b823b28","Type":"ContainerDied","Data":"0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d"}
Sep 09 21:07:01.721743 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.721691    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf" event={"ID":"4eb83949-b8fa-46d5-8fec-3d1a8b823b28","Type":"ContainerDied","Data":"4f958f42ef4f4b819b0795b610a11bd4796628a7aa717ba0e7ffc9245882db6e"}
Sep 09 21:07:01.721743 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.721723    2566 scope.go:117] "RemoveContainer" containerID="0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d"
Sep 09 21:07:01.731944 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.731927    2566 scope.go:117] "RemoveContainer" containerID="0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d"
Sep 09 21:07:01.732225 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:07:01.732203    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d\": container with ID starting with 0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d not found: ID does not exist" containerID="0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d"
Sep 09 21:07:01.732268 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.732234    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d"} err="failed to get container status \"0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d\": rpc error: code = NotFound desc = could not find container \"0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d\": container with ID starting with 0613e0f91177da69bbb53179d068b22701be80d00110891f877da9afa6ec742d not found: ID does not exist"
Sep 09 21:07:01.744842 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.744818    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"]
Sep 09 21:07:01.752334 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.752293    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-d36b91-6747cb4dd5-k4hnf"]
Sep 09 21:07:01.813971 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:01.813939    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:02.286579 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:02.286547    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4eb83949-b8fa-46d5-8fec-3d1a8b823b28" path="/var/lib/kubelet/pods/4eb83949-b8fa-46d5-8fec-3d1a8b823b28/volumes"
Sep 09 21:07:02.286966 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:02.286864    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9e9addb-0874-4848-bab8-325d88f63369" path="/var/lib/kubelet/pods/e9e9addb-0874-4848-bab8-325d88f63369/volumes"
Sep 09 21:07:05.255358 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.255293    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq" podUID="eec4de89-5aea-491b-849f-fb7351b9d9ff" containerName="main" containerID="cri-o://b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455" gracePeriod=45
Sep 09 21:07:05.262362 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.262330    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br" podUID="ddc04ede-7549-48d2-ae86-0e9472d263ca" containerName="main" containerID="cri-o://a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892" gracePeriod=45
Sep 09 21:07:05.280258 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.280224    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l" podUID="b3cd2bcc-7c04-4981-82f4-f21649ec76f3" containerName="main" containerID="cri-o://c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab" gracePeriod=45
Sep 09 21:07:05.293001 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.292969    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6" podUID="975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" containerName="main" containerID="cri-o://0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510" gracePeriod=45
Sep 09 21:07:05.577230 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.577185    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:07:05.597985 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.597955    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 21:07:05.653289 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.653262    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:07:05.655408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.655384    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 21:07:05.732607 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732573    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6c8g9\" (UniqueName: \"kubernetes.io/projected/eec4de89-5aea-491b-849f-fb7351b9d9ff-kube-api-access-6c8g9\") pod \"eec4de89-5aea-491b-849f-fb7351b9d9ff\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") "
Sep 09 21:07:05.732795 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732631    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-model-cache\") pod \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") "
Sep 09 21:07:05.732795 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732668    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q7t92\" (UniqueName: \"kubernetes.io/projected/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-kube-api-access-q7t92\") pod \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") "
Sep 09 21:07:05.732795 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-tmp-dir\") pod \"ddc04ede-7549-48d2-ae86-0e9472d263ca\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") "
Sep 09 21:07:05.732795 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732733    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-model-cache\") pod \"eec4de89-5aea-491b-849f-fb7351b9d9ff\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") "
Sep 09 21:07:05.732795 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732755    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4de89-5aea-491b-849f-fb7351b9d9ff-tls-certs\") pod \"eec4de89-5aea-491b-849f-fb7351b9d9ff\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") "
Sep 09 21:07:05.732795 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732784    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-dshm\") pod \"ddc04ede-7549-48d2-ae86-0e9472d263ca\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") "
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732828    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-dshm\") pod \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") "
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732864    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tmp-dir\") pod \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") "
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732886    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-home\") pod \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") "
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732953    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tmp-dir\") pod \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") "
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732948    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ddc04ede-7549-48d2-ae86-0e9472d263ca" (UID: "ddc04ede-7549-48d2-ae86-0e9472d263ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732981    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-dshm\") pod \"eec4de89-5aea-491b-849f-fb7351b9d9ff\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") "
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.732980    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-model-cache" (OuterVolumeSpecName: "model-cache") pod "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" (UID: "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733010    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6rxq\" (UniqueName: \"kubernetes.io/projected/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-kube-api-access-x6rxq\") pod \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") "
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733009    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-model-cache" (OuterVolumeSpecName: "model-cache") pod "eec4de89-5aea-491b-849f-fb7351b9d9ff" (UID: "eec4de89-5aea-491b-849f-fb7351b9d9ff"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733055    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5d2h\" (UniqueName: \"kubernetes.io/projected/ddc04ede-7549-48d2-ae86-0e9472d263ca-kube-api-access-t5d2h\") pod \"ddc04ede-7549-48d2-ae86-0e9472d263ca\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") "
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733086    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-tmp-dir\") pod \"eec4de89-5aea-491b-849f-fb7351b9d9ff\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") "
Sep 09 21:07:05.733118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733114    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-home\") pod \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") "
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733122    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b3cd2bcc-7c04-4981-82f4-f21649ec76f3" (UID: "b3cd2bcc-7c04-4981-82f4-f21649ec76f3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733181    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-dshm\") pod \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") "
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733202    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" (UID: "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733224    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tls-certs\") pod \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\" (UID: \"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11\") "
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733248    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-home\") pod \"eec4de89-5aea-491b-849f-fb7351b9d9ff\" (UID: \"eec4de89-5aea-491b-849f-fb7351b9d9ff\") "
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733286    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tls-certs\") pod \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") "
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733370    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-model-cache\") pod \"ddc04ede-7549-48d2-ae86-0e9472d263ca\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") "
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733410    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-model-cache\") pod \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\" (UID: \"b3cd2bcc-7c04-4981-82f4-f21649ec76f3\") "
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733431    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc04ede-7549-48d2-ae86-0e9472d263ca-tls-certs\") pod \"ddc04ede-7549-48d2-ae86-0e9472d263ca\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") "
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733463    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-home" (OuterVolumeSpecName: "home") pod "b3cd2bcc-7c04-4981-82f4-f21649ec76f3" (UID: "b3cd2bcc-7c04-4981-82f4-f21649ec76f3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.733757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733490    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-home\") pod \"ddc04ede-7549-48d2-ae86-0e9472d263ca\" (UID: \"ddc04ede-7549-48d2-ae86-0e9472d263ca\") "
Sep 09 21:07:05.734360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733911    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.734360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733932    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.734360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733945    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.734360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733958    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.734360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733970    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.734360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.733982    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.734360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.734184    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-home" (OuterVolumeSpecName: "home") pod "ddc04ede-7549-48d2-ae86-0e9472d263ca" (UID: "ddc04ede-7549-48d2-ae86-0e9472d263ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.734723 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.734372    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "eec4de89-5aea-491b-849f-fb7351b9d9ff" (UID: "eec4de89-5aea-491b-849f-fb7351b9d9ff"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.734723 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.734558    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-home" (OuterVolumeSpecName: "home") pod "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" (UID: "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.735579 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.735550    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-dshm" (OuterVolumeSpecName: "dshm") pod "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" (UID: "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.736233 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.735824    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eec4de89-5aea-491b-849f-fb7351b9d9ff-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "eec4de89-5aea-491b-849f-fb7351b9d9ff" (UID: "eec4de89-5aea-491b-849f-fb7351b9d9ff"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:05.736357 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.735869    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc04ede-7549-48d2-ae86-0e9472d263ca-kube-api-access-t5d2h" (OuterVolumeSpecName: "kube-api-access-t5d2h") pod "ddc04ede-7549-48d2-ae86-0e9472d263ca" (UID: "ddc04ede-7549-48d2-ae86-0e9472d263ca"). InnerVolumeSpecName "kube-api-access-t5d2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:05.736357 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.735998    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-kube-api-access-q7t92" (OuterVolumeSpecName: "kube-api-access-q7t92") pod "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" (UID: "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11"). InnerVolumeSpecName "kube-api-access-q7t92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:05.736357 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.736004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-dshm" (OuterVolumeSpecName: "dshm") pod "ddc04ede-7549-48d2-ae86-0e9472d263ca" (UID: "ddc04ede-7549-48d2-ae86-0e9472d263ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.736357 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.736232    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-home" (OuterVolumeSpecName: "home") pod "eec4de89-5aea-491b-849f-fb7351b9d9ff" (UID: "eec4de89-5aea-491b-849f-fb7351b9d9ff"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.736583 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.736441    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-model-cache" (OuterVolumeSpecName: "model-cache") pod "b3cd2bcc-7c04-4981-82f4-f21649ec76f3" (UID: "b3cd2bcc-7c04-4981-82f4-f21649ec76f3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.736654 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.736587    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-model-cache" (OuterVolumeSpecName: "model-cache") pod "ddc04ede-7549-48d2-ae86-0e9472d263ca" (UID: "ddc04ede-7549-48d2-ae86-0e9472d263ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.737025 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.736995    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eec4de89-5aea-491b-849f-fb7351b9d9ff-kube-api-access-6c8g9" (OuterVolumeSpecName: "kube-api-access-6c8g9") pod "eec4de89-5aea-491b-849f-fb7351b9d9ff" (UID: "eec4de89-5aea-491b-849f-fb7351b9d9ff"). InnerVolumeSpecName "kube-api-access-6c8g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:05.737509 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.737468    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-dshm" (OuterVolumeSpecName: "dshm") pod "eec4de89-5aea-491b-849f-fb7351b9d9ff" (UID: "eec4de89-5aea-491b-849f-fb7351b9d9ff"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.737664 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.737631    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-dshm" (OuterVolumeSpecName: "dshm") pod "b3cd2bcc-7c04-4981-82f4-f21649ec76f3" (UID: "b3cd2bcc-7c04-4981-82f4-f21649ec76f3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:07:05.737798 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.737781    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b3cd2bcc-7c04-4981-82f4-f21649ec76f3" (UID: "b3cd2bcc-7c04-4981-82f4-f21649ec76f3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:05.738253 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.738234    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-kube-api-access-x6rxq" (OuterVolumeSpecName: "kube-api-access-x6rxq") pod "b3cd2bcc-7c04-4981-82f4-f21649ec76f3" (UID: "b3cd2bcc-7c04-4981-82f4-f21649ec76f3"). InnerVolumeSpecName "kube-api-access-x6rxq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:07:05.738440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.738426    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" (UID: "975c6f12-0d20-44c1-8c4d-0b72a6cc5d11"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:05.738763 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.738750    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddc04ede-7549-48d2-ae86-0e9472d263ca-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ddc04ede-7549-48d2-ae86-0e9472d263ca" (UID: "ddc04ede-7549-48d2-ae86-0e9472d263ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:07:05.745952 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.745928    2566 generic.go:356] "Generic (PLEG): container finished" podID="975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" containerID="0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510" exitCode=0
Sep 09 21:07:05.746028 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.745966    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"
Sep 09 21:07:05.746028 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.746007    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6" event={"ID":"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11","Type":"ContainerDied","Data":"0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510"}
Sep 09 21:07:05.746147 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.746045    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6" event={"ID":"975c6f12-0d20-44c1-8c4d-0b72a6cc5d11","Type":"ContainerDied","Data":"c2278daf4d079ba911a9641198ab2ccb3ccfa9c08b17742fae31a1df08dc9a52"}
Sep 09 21:07:05.746147 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.746067    2566 scope.go:117] "RemoveContainer" containerID="0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510"
Sep 09 21:07:05.747480 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.747459    2566 generic.go:356] "Generic (PLEG): container finished" podID="eec4de89-5aea-491b-849f-fb7351b9d9ff" containerID="b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455" exitCode=0
Sep 09 21:07:05.747574 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.747500    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"
Sep 09 21:07:05.747574 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.747545    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq" event={"ID":"eec4de89-5aea-491b-849f-fb7351b9d9ff","Type":"ContainerDied","Data":"b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455"}
Sep 09 21:07:05.747694 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.747628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq" event={"ID":"eec4de89-5aea-491b-849f-fb7351b9d9ff","Type":"ContainerDied","Data":"a7a5d3ec4df9084e35a57747309692c9ed85bbb00fcb6b52ea2f83b44c5c1f3b"}
Sep 09 21:07:05.749087 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.749065    2566 generic.go:356] "Generic (PLEG): container finished" podID="ddc04ede-7549-48d2-ae86-0e9472d263ca" containerID="a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892" exitCode=0
Sep 09 21:07:05.749203 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.749097    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"
Sep 09 21:07:05.749203 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.749149    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br" event={"ID":"ddc04ede-7549-48d2-ae86-0e9472d263ca","Type":"ContainerDied","Data":"a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892"}
Sep 09 21:07:05.749203 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.749199    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br" event={"ID":"ddc04ede-7549-48d2-ae86-0e9472d263ca","Type":"ContainerDied","Data":"cf34c2bc013a846ac29277df720cc1c03ea04b7021c54c7d24134fb0d1c97433"}
Sep 09 21:07:05.750746 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.750723    2566 generic.go:356] "Generic (PLEG): container finished" podID="b3cd2bcc-7c04-4981-82f4-f21649ec76f3" containerID="c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab" exitCode=0
Sep 09 21:07:05.750861 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.750765    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l" event={"ID":"b3cd2bcc-7c04-4981-82f4-f21649ec76f3","Type":"ContainerDied","Data":"c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab"}
Sep 09 21:07:05.750861 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.750769    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"
Sep 09 21:07:05.750861 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.750787    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l" event={"ID":"b3cd2bcc-7c04-4981-82f4-f21649ec76f3","Type":"ContainerDied","Data":"1251ea9dc30bcd7b0c485d4a6e1d9f833cdb0c3e0fb261521862568958514451"}
Sep 09 21:07:05.756988 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.756970    2566 scope.go:117] "RemoveContainer" containerID="0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510"
Sep 09 21:07:05.757301 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:07:05.757274    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510\": container with ID starting with 0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510 not found: ID does not exist" containerID="0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510"
Sep 09 21:07:05.757405 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.757383    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510"} err="failed to get container status \"0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510\": rpc error: code = NotFound desc = could not find container \"0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510\": container with ID starting with 0c5e18052c5e4033cdd769dbf0319c75143ddbcbaa16eb830d1a0c8fc9c88510 not found: ID does not exist"
Sep 09 21:07:05.757475 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.757407    2566 scope.go:117] "RemoveContainer" containerID="b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455"
Sep 09 21:07:05.766666 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.766649    2566 scope.go:117] "RemoveContainer" containerID="b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455"
Sep 09 21:07:05.766920 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:07:05.766900    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455\": container with ID starting with b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455 not found: ID does not exist" containerID="b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455"
Sep 09 21:07:05.766990 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.766929    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455"} err="failed to get container status \"b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455\": rpc error: code = NotFound desc = could not find container \"b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455\": container with ID starting with b04f7e583e44848d65fda9698c734458fbfbc4581dad939dae2953ce1a71f455 not found: ID does not exist"
Sep 09 21:07:05.766990 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.766953    2566 scope.go:117] "RemoveContainer" containerID="a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892"
Sep 09 21:07:05.774760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.774741    2566 scope.go:117] "RemoveContainer" containerID="a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892"
Sep 09 21:07:05.775023 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:07:05.775004    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892\": container with ID starting with a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892 not found: ID does not exist" containerID="a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892"
Sep 09 21:07:05.775089 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.775029    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892"} err="failed to get container status \"a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892\": rpc error: code = NotFound desc = could not find container \"a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892\": container with ID starting with a32f9fee9cf41d788da77b54ea027761fb64cc6bedaf904e1c5f1f0fbb48b892 not found: ID does not exist"
Sep 09 21:07:05.775089 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.775045    2566 scope.go:117] "RemoveContainer" containerID="c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab"
Sep 09 21:07:05.776612 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.776592    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"]
Sep 09 21:07:05.781633 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.781612    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/unconfigured-w0-d36b91-kserve-54d4996655-f5zk6"]
Sep 09 21:07:05.782607 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.782588    2566 scope.go:117] "RemoveContainer" containerID="c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab"
Sep 09 21:07:05.782880 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:07:05.782862    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab\": container with ID starting with c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab not found: ID does not exist" containerID="c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab"
Sep 09 21:07:05.782934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.782887    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab"} err="failed to get container status \"c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab\": rpc error: code = NotFound desc = could not find container \"c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab\": container with ID starting with c7aa604e4a011eea8c358b67386291cf88e6e0d432c858a9312ded478e5788ab not found: ID does not exist"
Sep 09 21:07:05.800370 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.800343    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"]
Sep 09 21:07:05.804238 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.804217    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/embedding-w0-d36b91-kserve-868c865f7-pdkxq"]
Sep 09 21:07:05.819039 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.819015    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"]
Sep 09 21:07:05.828328 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.825600    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/simulator-w0-d36b91-kserve-5bbd8d94d4-mgw8l"]
Sep 09 21:07:05.835180 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835151    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835328 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835188    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835328 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835233    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6rxq\" (UniqueName: \"kubernetes.io/projected/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-kube-api-access-x6rxq\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835328 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835254    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5d2h\" (UniqueName: \"kubernetes.io/projected/ddc04ede-7549-48d2-ae86-0e9472d263ca-kube-api-access-t5d2h\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835328 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835275    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835328 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835292    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835328 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835318    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835328 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835325    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835338    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec4de89-5aea-491b-849f-fb7351b9d9ff-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835353    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835368    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835378    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b3cd2bcc-7c04-4981-82f4-f21649ec76f3-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835392    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ddc04ede-7549-48d2-ae86-0e9472d263ca-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835406    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835420    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c8g9\" (UniqueName: \"kubernetes.io/projected/eec4de89-5aea-491b-849f-fb7351b9d9ff-kube-api-access-6c8g9\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835433    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7t92\" (UniqueName: \"kubernetes.io/projected/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11-kube-api-access-q7t92\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835445    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec4de89-5aea-491b-849f-fb7351b9d9ff-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.835582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.835457    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ddc04ede-7549-48d2-ae86-0e9472d263ca-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:07:05.842418 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.842395    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"]
Sep 09 21:07:05.848206 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:05.848186    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-d36b91/premium-w0-d36b91-kserve-6bcf747c-qt6br"]
Sep 09 21:07:06.286477 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:06.286447    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" path="/var/lib/kubelet/pods/975c6f12-0d20-44c1-8c4d-0b72a6cc5d11/volumes"
Sep 09 21:07:06.286885 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:06.286871    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3cd2bcc-7c04-4981-82f4-f21649ec76f3" path="/var/lib/kubelet/pods/b3cd2bcc-7c04-4981-82f4-f21649ec76f3/volumes"
Sep 09 21:07:06.287221 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:06.287208    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddc04ede-7549-48d2-ae86-0e9472d263ca" path="/var/lib/kubelet/pods/ddc04ede-7549-48d2-ae86-0e9472d263ca/volumes"
Sep 09 21:07:06.287833 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:06.287819    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eec4de89-5aea-491b-849f-fb7351b9d9ff" path="/var/lib/kubelet/pods/eec4de89-5aea-491b-849f-fb7351b9d9ff/volumes"
Sep 09 21:07:07.702740 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:07.702711    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:07:08.227651 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:08.227607    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:08.332196 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:08.332143    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:08.588349 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:08.588299    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:18.484708 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:18.484627    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:20.466439 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:20.466408    2566 scope.go:117] "RemoveContainer" containerID="ebe95b11b3c52139a7abacbed4ccce7d08db8dfd232c9b0d806729403fcabd0b"
Sep 09 21:07:21.323549 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:21.323504    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:27.353269 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:27.353209    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:27.424614 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:27.424580    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:28.312771 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:28.312729    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:30.243541 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:30.243499    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:07:36.804815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:36.804765    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:36.906608 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:36.906571    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:37.032399 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:37.032359    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:37.127140 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:37.127050    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:37.220286 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:37.220247    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:37.464139 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:37.464053    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:41.947574 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:41.947540    2566 generic.go:356] "Generic (PLEG): container finished" podID="0d3e45df-f867-40f3-819e-db13c8b74726" containerID="069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751" exitCode=0
Sep 09 21:07:41.948003 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:41.947594    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" event={"ID":"0d3e45df-f867-40f3-819e-db13c8b74726","Type":"ContainerDied","Data":"069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751"}
Sep 09 21:07:41.948097 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:41.948078    2566 scope.go:117] "RemoveContainer" containerID="069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751"
Sep 09 21:07:41.949258 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:41.949228    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:41.949360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:41.949294    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:41.949360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:41.949346    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:07:42.954792 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:42.954760    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" event={"ID":"0d3e45df-f867-40f3-819e-db13c8b74726","Type":"ContainerStarted","Data":"9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5"}
Sep 09 21:07:43.394116 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:43.394075    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:07:43.394340 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:43.394274    2566 patch_prober.go:28] interesting pod/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 21:07:43.394397 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:43.394352    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 21:07:44.263914 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:44.263869    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:44.394400 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:44.394360    2566 patch_prober.go:28] interesting pod/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 21:07:44.394614 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:44.394436    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 21:07:44.602468 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:44.602383    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:44.729460 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:44.729425    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:45.164924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:45.164893    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:07:45.394413 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:45.394373    2566 patch_prober.go:28] interesting pod/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 21:07:45.394876 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:45.394430    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 21:07:46.394269 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:46.394229    2566 patch_prober.go:28] interesting pod/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 21:07:46.394448 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:46.394288    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 21:07:47.394688 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:47.394645    2566 patch_prober.go:28] interesting pod/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 21:07:47.395075 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:47.394711    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 21:07:48.394284 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:48.394246    2566 patch_prober.go:28] interesting pod/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused" start-of-body=
Sep 09 21:07:48.394486 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:48.394303    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.67:15021/healthz/ready\": dial tcp 10.132.0.67:15021: connect: connection refused"
Sep 09 21:07:49.398170 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:49.398139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:07:49.398643 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:49.398526    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:07:49.399157 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:49.399138    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:07:54.038287 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:07:54.038254    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:00.451159 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.451126    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:00.467749 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.467721    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"]
Sep 09 21:08:00.467980 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.467953    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn" podUID="d6879aeb-520d-45d2-a01c-1eff17166403" containerName="payload-processing" containerID="cri-o://542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283" gracePeriod=30
Sep 09 21:08:00.540851 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.540816    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:00.644990 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.644953    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:00.733171 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.733144    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:08:00.775352 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.775293    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6879aeb-520d-45d2-a01c-1eff17166403-plugins-config-volume\") pod \"d6879aeb-520d-45d2-a01c-1eff17166403\" (UID: \"d6879aeb-520d-45d2-a01c-1eff17166403\") "
Sep 09 21:08:00.775507 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.775371    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9nkv\" (UniqueName: \"kubernetes.io/projected/d6879aeb-520d-45d2-a01c-1eff17166403-kube-api-access-j9nkv\") pod \"d6879aeb-520d-45d2-a01c-1eff17166403\" (UID: \"d6879aeb-520d-45d2-a01c-1eff17166403\") "
Sep 09 21:08:00.775757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.775728    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6879aeb-520d-45d2-a01c-1eff17166403-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d6879aeb-520d-45d2-a01c-1eff17166403" (UID: "d6879aeb-520d-45d2-a01c-1eff17166403"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:00.777633 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.777608    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6879aeb-520d-45d2-a01c-1eff17166403-kube-api-access-j9nkv" (OuterVolumeSpecName: "kube-api-access-j9nkv") pod "d6879aeb-520d-45d2-a01c-1eff17166403" (UID: "d6879aeb-520d-45d2-a01c-1eff17166403"). InnerVolumeSpecName "kube-api-access-j9nkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:00.876204 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.876165    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6879aeb-520d-45d2-a01c-1eff17166403-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:00.876204 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.876201    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9nkv\" (UniqueName: \"kubernetes.io/projected/d6879aeb-520d-45d2-a01c-1eff17166403-kube-api-access-j9nkv\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:00.949844 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:00.949809    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:01.050838 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.050805    2566 generic.go:356] "Generic (PLEG): container finished" podID="d6879aeb-520d-45d2-a01c-1eff17166403" containerID="542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283" exitCode=0
Sep 09 21:08:01.051023 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.050849    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"
Sep 09 21:08:01.051023 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.050894    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn" event={"ID":"d6879aeb-520d-45d2-a01c-1eff17166403","Type":"ContainerDied","Data":"542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283"}
Sep 09 21:08:01.051023 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.050960    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn" event={"ID":"d6879aeb-520d-45d2-a01c-1eff17166403","Type":"ContainerDied","Data":"b80445788b49c38bf3285c80bb665b42d7bd006df8c3106ae31ce55edb2dea76"}
Sep 09 21:08:01.051023 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.050983    2566 scope.go:117] "RemoveContainer" containerID="542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283"
Sep 09 21:08:01.060004 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.059987    2566 scope.go:117] "RemoveContainer" containerID="542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283"
Sep 09 21:08:01.060264 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:08:01.060246    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283\": container with ID starting with 542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283 not found: ID does not exist" containerID="542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283"
Sep 09 21:08:01.060340 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.060273    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283"} err="failed to get container status \"542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283\": rpc error: code = NotFound desc = could not find container \"542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283\": container with ID starting with 542f0f6fb5381946906019457f41928650ebbb7268f0b5ff96fbb545ae8da283 not found: ID does not exist"
Sep 09 21:08:01.076886 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.076862    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"]
Sep 09 21:08:01.080503 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.080482    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-0622e2-6d9748996c-xrwrn"]
Sep 09 21:08:01.169943 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.169914    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:01.575086 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:01.575051    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:08:02.286827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:02.286798    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6879aeb-520d-45d2-a01c-1eff17166403" path="/var/lib/kubelet/pods/d6879aeb-520d-45d2-a01c-1eff17166403/volumes"
Sep 09 21:08:07.533926 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:07.533872    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:07.684732 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:07.684699    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:07.770199 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:07.770168    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"]
Sep 09 21:08:07.770453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:07.770429    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy" containerID="cri-o://9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5" gracePeriod=30
Sep 09 21:08:07.838098 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:07.838016    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:08.323469 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:08.323433    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:13.022358 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.022335    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:08:13.081838 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.081762    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e45df-f867-40f3-819e-db13c8b74726-istio-podinfo\") pod \"0d3e45df-f867-40f3-819e-db13c8b74726\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") "
Sep 09 21:08:13.081838 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.081797    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-credential-socket\") pod \"0d3e45df-f867-40f3-819e-db13c8b74726\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") "
Sep 09 21:08:13.081838 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.081822    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-envoy\") pod \"0d3e45df-f867-40f3-819e-db13c8b74726\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") "
Sep 09 21:08:13.082102 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.081841    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e45df-f867-40f3-819e-db13c8b74726-istiod-ca-cert\") pod \"0d3e45df-f867-40f3-819e-db13c8b74726\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") "
Sep 09 21:08:13.082102 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.081971    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-data\") pod \"0d3e45df-f867-40f3-819e-db13c8b74726\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") "
Sep 09 21:08:13.082102 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082076    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-certs\") pod \"0d3e45df-f867-40f3-819e-db13c8b74726\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") "
Sep 09 21:08:13.082102 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082090    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0d3e45df-f867-40f3-819e-db13c8b74726" (UID: "0d3e45df-f867-40f3-819e-db13c8b74726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:13.082350 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082117    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nvgkt\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-kube-api-access-nvgkt\") pod \"0d3e45df-f867-40f3-819e-db13c8b74726\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") "
Sep 09 21:08:13.082350 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082168    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-istio-token\") pod \"0d3e45df-f867-40f3-819e-db13c8b74726\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") "
Sep 09 21:08:13.082350 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082208    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-socket\") pod \"0d3e45df-f867-40f3-819e-db13c8b74726\" (UID: \"0d3e45df-f867-40f3-819e-db13c8b74726\") "
Sep 09 21:08:13.082350 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082206    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d3e45df-f867-40f3-819e-db13c8b74726-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0d3e45df-f867-40f3-819e-db13c8b74726" (UID: "0d3e45df-f867-40f3-819e-db13c8b74726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:13.082350 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082223    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-data" (OuterVolumeSpecName: "istio-data") pod "0d3e45df-f867-40f3-819e-db13c8b74726" (UID: "0d3e45df-f867-40f3-819e-db13c8b74726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:13.082350 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0d3e45df-f867-40f3-819e-db13c8b74726" (UID: "0d3e45df-f867-40f3-819e-db13c8b74726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:13.082658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082487    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0d3e45df-f867-40f3-819e-db13c8b74726" (UID: "0d3e45df-f867-40f3-819e-db13c8b74726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:13.082658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082586    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:13.082658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082604    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-workload-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:13.082658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082618    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-credential-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:13.082658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082632    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d3e45df-f867-40f3-819e-db13c8b74726-istiod-ca-cert\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:13.082658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.082644    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-data\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:13.084096 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.084070    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d3e45df-f867-40f3-819e-db13c8b74726-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0d3e45df-f867-40f3-819e-db13c8b74726" (UID: "0d3e45df-f867-40f3-819e-db13c8b74726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:08:13.084210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.084074    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0d3e45df-f867-40f3-819e-db13c8b74726" (UID: "0d3e45df-f867-40f3-819e-db13c8b74726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:08:13.084210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.084158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-istio-token" (OuterVolumeSpecName: "istio-token") pod "0d3e45df-f867-40f3-819e-db13c8b74726" (UID: "0d3e45df-f867-40f3-819e-db13c8b74726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:13.084210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.084193    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-kube-api-access-nvgkt" (OuterVolumeSpecName: "kube-api-access-nvgkt") pod "0d3e45df-f867-40f3-819e-db13c8b74726" (UID: "0d3e45df-f867-40f3-819e-db13c8b74726"). InnerVolumeSpecName "kube-api-access-nvgkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:13.113912 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.113882    2566 generic.go:356] "Generic (PLEG): container finished" podID="0d3e45df-f867-40f3-819e-db13c8b74726" containerID="9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5" exitCode=0
Sep 09 21:08:13.114029 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.113934    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"
Sep 09 21:08:13.114029 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.113934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" event={"ID":"0d3e45df-f867-40f3-819e-db13c8b74726","Type":"ContainerDied","Data":"9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5"}
Sep 09 21:08:13.114109 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.114056    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5" event={"ID":"0d3e45df-f867-40f3-819e-db13c8b74726","Type":"ContainerDied","Data":"ce3534e15b489de313639c0483afbba2d705eafcb57501b1baa18db81798e813"}
Sep 09 21:08:13.114109 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.114087    2566 scope.go:117] "RemoveContainer" containerID="9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5"
Sep 09 21:08:13.122787 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.122771    2566 scope.go:117] "RemoveContainer" containerID="069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751"
Sep 09 21:08:13.130237 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.130219    2566 scope.go:117] "RemoveContainer" containerID="9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5"
Sep 09 21:08:13.130525 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:08:13.130506    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5\": container with ID starting with 9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5 not found: ID does not exist" containerID="9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5"
Sep 09 21:08:13.130600 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.130534    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5"} err="failed to get container status \"9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5\": rpc error: code = NotFound desc = could not find container \"9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5\": container with ID starting with 9fc81cd28aa4fd5a4504536c6dbc1327ef3f371c52fefa3ee6d04034836462a5 not found: ID does not exist"
Sep 09 21:08:13.130600 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.130550    2566 scope.go:117] "RemoveContainer" containerID="069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751"
Sep 09 21:08:13.130797 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:08:13.130781    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751\": container with ID starting with 069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751 not found: ID does not exist" containerID="069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751"
Sep 09 21:08:13.130841 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.130800    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751"} err="failed to get container status \"069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751\": rpc error: code = NotFound desc = could not find container \"069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751\": container with ID starting with 069afab16539154b9d28442d5148b6dcbb80124ac0e15d19f4ba09d456a63751 not found: ID does not exist"
Sep 09 21:08:13.137092 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.137071    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"]
Sep 09 21:08:13.142569 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.142547    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-0622e2-openshift-default-786bc9d79b-87tk5"]
Sep 09 21:08:13.183693 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.183669    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvgkt\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-kube-api-access-nvgkt\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:13.183778 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.183691    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d3e45df-f867-40f3-819e-db13c8b74726-istio-token\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:13.183778 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.183704    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d3e45df-f867-40f3-819e-db13c8b74726-istio-podinfo\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:13.183778 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:13.183714    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d3e45df-f867-40f3-819e-db13c8b74726-istio-envoy\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:14.286029 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:14.285993    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" path="/var/lib/kubelet/pods/0d3e45df-f867-40f3-819e-db13c8b74726/volumes"
Sep 09 21:08:18.938453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:18.938416    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:19.088659 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:19.088625    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:19.220913 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:19.220813    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:19.355736 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:19.355701    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:19.509250 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:19.509222    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:19.694446 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:19.694415    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:25.538041 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:25.538002    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:25.785148 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:25.785120    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:25.888354 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:25.888246    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:26.184656 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:26.184567    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:26.514784 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:26.514757    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:34.771487 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:34.771450    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:08:37.296366 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.296327    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:37.311650 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.311620    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"]
Sep 09 21:08:37.311826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.311803    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs" podUID="5b4f53bb-bf35-4a43-adc4-af50067f87bf" containerName="payload-pre-processing" containerID="cri-o://6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651" gracePeriod=30
Sep 09 21:08:37.395605 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.395569    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:37.514687 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.514636    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:37.576861 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.576840    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:08:37.607776 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.607742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:37.709851 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.709821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4f53bb-bf35-4a43-adc4-af50067f87bf-plugins-config-volume\") pod \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\" (UID: \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\") "
Sep 09 21:08:37.710011 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.709868    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bsk4g\" (UniqueName: \"kubernetes.io/projected/5b4f53bb-bf35-4a43-adc4-af50067f87bf-kube-api-access-bsk4g\") pod \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\" (UID: \"5b4f53bb-bf35-4a43-adc4-af50067f87bf\") "
Sep 09 21:08:37.710245 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.710219    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b4f53bb-bf35-4a43-adc4-af50067f87bf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b4f53bb-bf35-4a43-adc4-af50067f87bf" (UID: "5b4f53bb-bf35-4a43-adc4-af50067f87bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:08:37.711928 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.711909    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b4f53bb-bf35-4a43-adc4-af50067f87bf-kube-api-access-bsk4g" (OuterVolumeSpecName: "kube-api-access-bsk4g") pod "5b4f53bb-bf35-4a43-adc4-af50067f87bf" (UID: "5b4f53bb-bf35-4a43-adc4-af50067f87bf"). InnerVolumeSpecName "kube-api-access-bsk4g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:08:37.811476 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.811406    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b4f53bb-bf35-4a43-adc4-af50067f87bf-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:37.811476 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.811431    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsk4g\" (UniqueName: \"kubernetes.io/projected/5b4f53bb-bf35-4a43-adc4-af50067f87bf-kube-api-access-bsk4g\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:08:37.948585 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:37.948546    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:38.247781 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.247747    2566 generic.go:356] "Generic (PLEG): container finished" podID="5b4f53bb-bf35-4a43-adc4-af50067f87bf" containerID="6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651" exitCode=0
Sep 09 21:08:38.247938 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.247794    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"
Sep 09 21:08:38.247938 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.247817    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs" event={"ID":"5b4f53bb-bf35-4a43-adc4-af50067f87bf","Type":"ContainerDied","Data":"6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651"}
Sep 09 21:08:38.247938 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.247868    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs" event={"ID":"5b4f53bb-bf35-4a43-adc4-af50067f87bf","Type":"ContainerDied","Data":"e9409a46c88f0dd04d25f3f35fc731d846afc8dd8f3c439c6f5468b2d24946a8"}
Sep 09 21:08:38.247938 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.247899    2566 scope.go:117] "RemoveContainer" containerID="6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651"
Sep 09 21:08:38.256927 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.256907    2566 scope.go:117] "RemoveContainer" containerID="6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651"
Sep 09 21:08:38.257185 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:08:38.257165    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651\": container with ID starting with 6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651 not found: ID does not exist" containerID="6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651"
Sep 09 21:08:38.257257 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.257198    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651"} err="failed to get container status \"6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651\": rpc error: code = NotFound desc = could not find container \"6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651\": container with ID starting with 6fe7f5d891fea5c97dd69e6140a6c97eee3d0e8a294ad43c014f474a4c825651 not found: ID does not exist"
Sep 09 21:08:38.272346 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.272322    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"]
Sep 09 21:08:38.277890 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.277871    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-06e1ff-7fcb75bf8-4qmjs"]
Sep 09 21:08:38.285673 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:38.285649    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b4f53bb-bf35-4a43-adc4-af50067f87bf" path="/var/lib/kubelet/pods/5b4f53bb-bf35-4a43-adc4-af50067f87bf/volumes"
Sep 09 21:08:44.578191 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:44.578110    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:44.744681 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:44.744646    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:44.873714 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:44.873627    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:08:46.451476 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:08:46.451442    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:00.367709 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:00.367671    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:09:06.524411 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:06.524372    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:06.930269 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:06.930182    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:11.504950 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:11.504910    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"]
Sep 09 21:09:11.536026 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:11.535986    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"]
Sep 09 21:09:11.641879 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:11.641843    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:22.540526 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.540496    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"]
Sep 09 21:09:22.540987 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.540694    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb" podUID="4d2b5add-ae0d-425f-8f97-256709aa7865" containerName="maas-api" containerID="cri-o://bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54" gracePeriod=30
Sep 09 21:09:22.570256 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.570220    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:22.632187 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.632154    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"]
Sep 09 21:09:22.632446 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.632399    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl" podUID="82ddfb64-e9be-417a-91cf-fc6257a76f95" containerName="payload-processing" containerID="cri-o://ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b" gracePeriod=30
Sep 09 21:09:22.685769 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.685742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:22.783908 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.783869    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:22.887252 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.887221    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:22.887422 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.887403    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 21:09:22.914344 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.914302    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 21:09:22.975599 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:22.975564    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:23.033555 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.033525    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-maas-api-tls\") pod \"4d2b5add-ae0d-425f-8f97-256709aa7865\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") "
Sep 09 21:09:23.033698 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.033587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b7d2z\" (UniqueName: \"kubernetes.io/projected/82ddfb64-e9be-417a-91cf-fc6257a76f95-kube-api-access-b7d2z\") pod \"82ddfb64-e9be-417a-91cf-fc6257a76f95\" (UID: \"82ddfb64-e9be-417a-91cf-fc6257a76f95\") "
Sep 09 21:09:23.033698 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.033637    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm4nr\" (UniqueName: \"kubernetes.io/projected/4d2b5add-ae0d-425f-8f97-256709aa7865-kube-api-access-wm4nr\") pod \"4d2b5add-ae0d-425f-8f97-256709aa7865\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") "
Sep 09 21:09:23.033698 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.033676    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2b5add-ae0d-425f-8f97-256709aa7865-openshift-service-ca\") pod \"4d2b5add-ae0d-425f-8f97-256709aa7865\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") "
Sep 09 21:09:23.033822 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.033698    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-metrics-certs\") pod \"4d2b5add-ae0d-425f-8f97-256709aa7865\" (UID: \"4d2b5add-ae0d-425f-8f97-256709aa7865\") "
Sep 09 21:09:23.033822 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.033765    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ddfb64-e9be-417a-91cf-fc6257a76f95-plugins-config-volume\") pod \"82ddfb64-e9be-417a-91cf-fc6257a76f95\" (UID: \"82ddfb64-e9be-417a-91cf-fc6257a76f95\") "
Sep 09 21:09:23.034036 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.034004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d2b5add-ae0d-425f-8f97-256709aa7865-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4d2b5add-ae0d-425f-8f97-256709aa7865" (UID: "4d2b5add-ae0d-425f-8f97-256709aa7865"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:23.034241 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.034213    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ddfb64-e9be-417a-91cf-fc6257a76f95-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "82ddfb64-e9be-417a-91cf-fc6257a76f95" (UID: "82ddfb64-e9be-417a-91cf-fc6257a76f95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:09:23.035935 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.035912    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4d2b5add-ae0d-425f-8f97-256709aa7865" (UID: "4d2b5add-ae0d-425f-8f97-256709aa7865"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:23.036017 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.035965    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d2b5add-ae0d-425f-8f97-256709aa7865-kube-api-access-wm4nr" (OuterVolumeSpecName: "kube-api-access-wm4nr") pod "4d2b5add-ae0d-425f-8f97-256709aa7865" (UID: "4d2b5add-ae0d-425f-8f97-256709aa7865"). InnerVolumeSpecName "kube-api-access-wm4nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:23.036153 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.036132    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4d2b5add-ae0d-425f-8f97-256709aa7865" (UID: "4d2b5add-ae0d-425f-8f97-256709aa7865"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:23.036197 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.036140    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ddfb64-e9be-417a-91cf-fc6257a76f95-kube-api-access-b7d2z" (OuterVolumeSpecName: "kube-api-access-b7d2z") pod "82ddfb64-e9be-417a-91cf-fc6257a76f95" (UID: "82ddfb64-e9be-417a-91cf-fc6257a76f95"). InnerVolumeSpecName "kube-api-access-b7d2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:23.134860 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.134781    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82ddfb64-e9be-417a-91cf-fc6257a76f95-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:23.134860 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.134812    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-maas-api-tls\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:23.134860 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.134822    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7d2z\" (UniqueName: \"kubernetes.io/projected/82ddfb64-e9be-417a-91cf-fc6257a76f95-kube-api-access-b7d2z\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:23.134860 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.134833    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm4nr\" (UniqueName: \"kubernetes.io/projected/4d2b5add-ae0d-425f-8f97-256709aa7865-kube-api-access-wm4nr\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:23.134860 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.134842    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d2b5add-ae0d-425f-8f97-256709aa7865-openshift-service-ca\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:23.134860 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.134852    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d2b5add-ae0d-425f-8f97-256709aa7865-metrics-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:23.265383 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.265351    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:23.485785 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.485751    2566 generic.go:356] "Generic (PLEG): container finished" podID="4d2b5add-ae0d-425f-8f97-256709aa7865" containerID="bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54" exitCode=0
Sep 09 21:09:23.485969 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.485800    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"
Sep 09 21:09:23.485969 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.485841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb" event={"ID":"4d2b5add-ae0d-425f-8f97-256709aa7865","Type":"ContainerDied","Data":"bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54"}
Sep 09 21:09:23.485969 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.485902    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb" event={"ID":"4d2b5add-ae0d-425f-8f97-256709aa7865","Type":"ContainerDied","Data":"c2436fc55d9842375fd260f347417690a281c39df12347d3d7c92a4742660b25"}
Sep 09 21:09:23.485969 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.485933    2566 scope.go:117] "RemoveContainer" containerID="bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54"
Sep 09 21:09:23.487284 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.487263    2566 generic.go:356] "Generic (PLEG): container finished" podID="82ddfb64-e9be-417a-91cf-fc6257a76f95" containerID="ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b" exitCode=0
Sep 09 21:09:23.487400 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.487361    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"
Sep 09 21:09:23.487457 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.487395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl" event={"ID":"82ddfb64-e9be-417a-91cf-fc6257a76f95","Type":"ContainerDied","Data":"ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b"}
Sep 09 21:09:23.487457 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.487429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl" event={"ID":"82ddfb64-e9be-417a-91cf-fc6257a76f95","Type":"ContainerDied","Data":"0706d0c25618458d3f5be5518e778418cdbb29381d3c91faae896c5dd5e99f44"}
Sep 09 21:09:23.496081 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.496060    2566 scope.go:117] "RemoveContainer" containerID="bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54"
Sep 09 21:09:23.496384 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:09:23.496362    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54\": container with ID starting with bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54 not found: ID does not exist" containerID="bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54"
Sep 09 21:09:23.496441 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.496391    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54"} err="failed to get container status \"bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54\": rpc error: code = NotFound desc = could not find container \"bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54\": container with ID starting with bae72fd6032d2ec30aeb11a1ebcf26a13f42748a5f1b98a923ab6a3a64a00e54 not found: ID does not exist"
Sep 09 21:09:23.496441 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.496411    2566 scope.go:117] "RemoveContainer" containerID="ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b"
Sep 09 21:09:23.504022 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.504006    2566 scope.go:117] "RemoveContainer" containerID="ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b"
Sep 09 21:09:23.504258 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:09:23.504241    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b\": container with ID starting with ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b not found: ID does not exist" containerID="ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b"
Sep 09 21:09:23.504330 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.504264    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b"} err="failed to get container status \"ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b\": rpc error: code = NotFound desc = could not find container \"ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b\": container with ID starting with ccf47ceea458ba44073fc7ed6cf2983359123d73ca26a29c2b2e861ec9b5064b not found: ID does not exist"
Sep 09 21:09:23.520671 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.520648    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"]
Sep 09 21:09:23.526824 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.526802    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-47e41c-6bff66ddcb-5l6bb"]
Sep 09 21:09:23.555863 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.555838    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"]
Sep 09 21:09:23.560406 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:23.560388    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-47e41c-5d75b77d59-hzmhl"]
Sep 09 21:09:24.072499 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:24.072466    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:09:24.286149 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:24.286118    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d2b5add-ae0d-425f-8f97-256709aa7865" path="/var/lib/kubelet/pods/4d2b5add-ae0d-425f-8f97-256709aa7865/volumes"
Sep 09 21:09:24.286509 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:24.286495    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="82ddfb64-e9be-417a-91cf-fc6257a76f95" path="/var/lib/kubelet/pods/82ddfb64-e9be-417a-91cf-fc6257a76f95/volumes"
Sep 09 21:09:26.524871 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.524811    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm" podUID="27c16589-1774-4286-b956-2324aeb5101d" containerName="main" containerID="cri-o://ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25" gracePeriod=45
Sep 09 21:09:26.555164 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.555129    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts" podUID="ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" containerName="main" containerID="cri-o://533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a" gracePeriod=45
Sep 09 21:09:26.827285 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.827258    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 21:09:26.829925 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.829902    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 21:09:26.972580 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972548    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-home\") pod \"27c16589-1774-4286-b956-2324aeb5101d\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") "
Sep 09 21:09:26.972744 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972605    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7sr6b\" (UniqueName: \"kubernetes.io/projected/27c16589-1774-4286-b956-2324aeb5101d-kube-api-access-7sr6b\") pod \"27c16589-1774-4286-b956-2324aeb5101d\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") "
Sep 09 21:09:26.972744 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972639    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tmp-dir\") pod \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") "
Sep 09 21:09:26.972744 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972685    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-tmp-dir\") pod \"27c16589-1774-4286-b956-2324aeb5101d\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") "
Sep 09 21:09:26.972744 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972713    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-home\") pod \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") "
Sep 09 21:09:26.972934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972763    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c16589-1774-4286-b956-2324aeb5101d-tls-certs\") pod \"27c16589-1774-4286-b956-2324aeb5101d\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") "
Sep 09 21:09:26.972934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-dshm\") pod \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") "
Sep 09 21:09:26.972934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972852    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8xrns\" (UniqueName: \"kubernetes.io/projected/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-kube-api-access-8xrns\") pod \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") "
Sep 09 21:09:26.972934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972886    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-model-cache\") pod \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") "
Sep 09 21:09:26.972934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tls-certs\") pod \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\" (UID: \"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d\") "
Sep 09 21:09:26.973210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972819    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-home" (OuterVolumeSpecName: "home") pod "27c16589-1774-4286-b956-2324aeb5101d" (UID: "27c16589-1774-4286-b956-2324aeb5101d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:26.973210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972971    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-dshm\") pod \"27c16589-1774-4286-b956-2324aeb5101d\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") "
Sep 09 21:09:26.973210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "27c16589-1774-4286-b956-2324aeb5101d" (UID: "27c16589-1774-4286-b956-2324aeb5101d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:26.973210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972928    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" (UID: "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:26.973210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.972989    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-home" (OuterVolumeSpecName: "home") pod "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" (UID: "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:26.973210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.973013    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-model-cache\") pod \"27c16589-1774-4286-b956-2324aeb5101d\" (UID: \"27c16589-1774-4286-b956-2324aeb5101d\") "
Sep 09 21:09:26.973210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.973155    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" (UID: "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:26.973627 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.973380    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-model-cache" (OuterVolumeSpecName: "model-cache") pod "27c16589-1774-4286-b956-2324aeb5101d" (UID: "27c16589-1774-4286-b956-2324aeb5101d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:26.973627 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.973394    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:26.973627 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.973415    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:26.973627 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.973431    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:26.973627 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.973468    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:26.973627 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.973492    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:26.975008 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.974977    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27c16589-1774-4286-b956-2324aeb5101d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "27c16589-1774-4286-b956-2324aeb5101d" (UID: "27c16589-1774-4286-b956-2324aeb5101d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:26.975132 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.975059    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" (UID: "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:09:26.975271 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.975249    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c16589-1774-4286-b956-2324aeb5101d-kube-api-access-7sr6b" (OuterVolumeSpecName: "kube-api-access-7sr6b") pod "27c16589-1774-4286-b956-2324aeb5101d" (UID: "27c16589-1774-4286-b956-2324aeb5101d"). InnerVolumeSpecName "kube-api-access-7sr6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:26.975425 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.975327    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-kube-api-access-8xrns" (OuterVolumeSpecName: "kube-api-access-8xrns") pod "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" (UID: "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d"). InnerVolumeSpecName "kube-api-access-8xrns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:09:26.975496 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.975447    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-dshm" (OuterVolumeSpecName: "dshm") pod "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" (UID: "ef6dc0d7-2992-4ff4-adff-bb6d66c9331d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:26.975554 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:26.975508    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-dshm" (OuterVolumeSpecName: "dshm") pod "27c16589-1774-4286-b956-2324aeb5101d" (UID: "27c16589-1774-4286-b956-2324aeb5101d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:09:27.074290 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.074218    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:27.074290 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.074245    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:27.074290 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.074254    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/27c16589-1774-4286-b956-2324aeb5101d-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:27.074290 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.074263    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sr6b\" (UniqueName: \"kubernetes.io/projected/27c16589-1774-4286-b956-2324aeb5101d-kube-api-access-7sr6b\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:27.074290 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.074273    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/27c16589-1774-4286-b956-2324aeb5101d-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:27.074290 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.074281    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:27.074290 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.074289    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xrns\" (UniqueName: \"kubernetes.io/projected/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d-kube-api-access-8xrns\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:09:27.510558 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.510527    2566 generic.go:356] "Generic (PLEG): container finished" podID="27c16589-1774-4286-b956-2324aeb5101d" containerID="ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25" exitCode=0
Sep 09 21:09:27.510735 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.510565    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"
Sep 09 21:09:27.510735 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.510604    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm" event={"ID":"27c16589-1774-4286-b956-2324aeb5101d","Type":"ContainerDied","Data":"ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25"}
Sep 09 21:09:27.510735 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.510642    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm" event={"ID":"27c16589-1774-4286-b956-2324aeb5101d","Type":"ContainerDied","Data":"56d6f8eb417ca4c144bd036c2e7cff5cca028628de5974a89fff09da5d14f6ed"}
Sep 09 21:09:27.510735 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.510659    2566 scope.go:117] "RemoveContainer" containerID="ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25"
Sep 09 21:09:27.511993 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.511957    2566 generic.go:356] "Generic (PLEG): container finished" podID="ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" containerID="533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a" exitCode=0
Sep 09 21:09:27.512117 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.511994    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts" event={"ID":"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d","Type":"ContainerDied","Data":"533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a"}
Sep 09 21:09:27.512117 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.512015    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"
Sep 09 21:09:27.512117 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.512028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts" event={"ID":"ef6dc0d7-2992-4ff4-adff-bb6d66c9331d","Type":"ContainerDied","Data":"e86272c4be2e06793538a33e1829c9ab74d491d2789961df5fbff26c51e5d76d"}
Sep 09 21:09:27.520046 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.519934    2566 scope.go:117] "RemoveContainer" containerID="ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25"
Sep 09 21:09:27.520257 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:09:27.520211    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25\": container with ID starting with ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25 not found: ID does not exist" containerID="ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25"
Sep 09 21:09:27.520257 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.520236    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25"} err="failed to get container status \"ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25\": rpc error: code = NotFound desc = could not find container \"ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25\": container with ID starting with ac9b5e53f0d3a230c15da887dee6835a77eb8bd7a0784a0ca71adc3bae57dc25 not found: ID does not exist"
Sep 09 21:09:27.520257 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.520253    2566 scope.go:117] "RemoveContainer" containerID="533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a"
Sep 09 21:09:27.528635 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.528425    2566 scope.go:117] "RemoveContainer" containerID="533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a"
Sep 09 21:09:27.528911 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:09:27.528679    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a\": container with ID starting with 533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a not found: ID does not exist" containerID="533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a"
Sep 09 21:09:27.528911 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.528698    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a"} err="failed to get container status \"533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a\": rpc error: code = NotFound desc = could not find container \"533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a\": container with ID starting with 533f2c92f116080419112e7eee5c3cb937af5e1a54a6646bb3dfdb51e42a393a not found: ID does not exist"
Sep 09 21:09:27.535787 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.535764    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"]
Sep 09 21:09:27.542740 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.542719    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/simulator-w4-47e41c-kserve-6f6c5dfc85-bc2ts"]
Sep 09 21:09:27.557809 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.557767    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"]
Sep 09 21:09:27.561800 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:27.561769    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-47e41c/distinct-w4-47e41c-kserve-85588d6967-mmnfm"]
Sep 09 21:09:28.288375 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:28.288346    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27c16589-1774-4286-b956-2324aeb5101d" path="/var/lib/kubelet/pods/27c16589-1774-4286-b956-2324aeb5101d/volumes"
Sep 09 21:09:28.288757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:28.288744    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" path="/var/lib/kubelet/pods/ef6dc0d7-2992-4ff4-adff-bb6d66c9331d/volumes"
Sep 09 21:09:29.932710 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:29.932678    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:30.029779 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:30.029742    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:30.257103 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:30.257068    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:42.682274 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:42.682237    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:44.734287 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:44.734255    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"]
Sep 09 21:09:44.770256 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:44.770219    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:46.366528 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:46.366498    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:09:47.774803 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:47.774763    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:47.903007 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:47.902965    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:50.300366 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:50.300330    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:54.497754 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.497720    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"]
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498047    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27c16589-1774-4286-b956-2324aeb5101d" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498058    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="27c16589-1774-4286-b956-2324aeb5101d" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498075    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eec4de89-5aea-491b-849f-fb7351b9d9ff" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498081    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="eec4de89-5aea-491b-849f-fb7351b9d9ff" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498089    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498107    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498121    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d2b5add-ae0d-425f-8f97-256709aa7865" containerName="maas-api"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498128    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d2b5add-ae0d-425f-8f97-256709aa7865" containerName="maas-api"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498134    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b4f53bb-bf35-4a43-adc4-af50067f87bf" containerName="payload-pre-processing"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498140    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b4f53bb-bf35-4a43-adc4-af50067f87bf" containerName="payload-pre-processing"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498147    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498152    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498161    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="82ddfb64-e9be-417a-91cf-fc6257a76f95" containerName="payload-processing"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498167    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="82ddfb64-e9be-417a-91cf-fc6257a76f95" containerName="payload-processing"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498174    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498180    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498187    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddc04ede-7549-48d2-ae86-0e9472d263ca" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498192    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddc04ede-7549-48d2-ae86-0e9472d263ca" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498199    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4eb83949-b8fa-46d5-8fec-3d1a8b823b28" containerName="payload-processing"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498204    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4eb83949-b8fa-46d5-8fec-3d1a8b823b28" containerName="payload-processing"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498211    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6879aeb-520d-45d2-a01c-1eff17166403" containerName="payload-processing"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498216    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6879aeb-520d-45d2-a01c-1eff17166403" containerName="payload-processing"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498221    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3cd2bcc-7c04-4981-82f4-f21649ec76f3" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498227    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3cd2bcc-7c04-4981-82f4-f21649ec76f3" containerName="main"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498234    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy"
Sep 09 21:09:54.498223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498239    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498292    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498298    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="82ddfb64-e9be-417a-91cf-fc6257a76f95" containerName="payload-processing"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498318    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6879aeb-520d-45d2-a01c-1eff17166403" containerName="payload-processing"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498328    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d2b5add-ae0d-425f-8f97-256709aa7865" containerName="maas-api"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498336    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="975c6f12-0d20-44c1-8c4d-0b72a6cc5d11" containerName="main"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498346    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="27c16589-1774-4286-b956-2324aeb5101d" containerName="main"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498355    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="eec4de89-5aea-491b-849f-fb7351b9d9ff" containerName="main"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498362    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0d3e45df-f867-40f3-819e-db13c8b74726" containerName="istio-proxy"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498370    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef6dc0d7-2992-4ff4-adff-bb6d66c9331d" containerName="main"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498379    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4eb83949-b8fa-46d5-8fec-3d1a8b823b28" containerName="payload-processing"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498384    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3cd2bcc-7c04-4981-82f4-f21649ec76f3" containerName="main"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498390    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddc04ede-7549-48d2-ae86-0e9472d263ca" containerName="main"
Sep 09 21:09:54.499095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.498394    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b4f53bb-bf35-4a43-adc4-af50067f87bf" containerName="payload-pre-processing"
Sep 09 21:09:54.501379 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.501362    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.504679 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.504652    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-615c6f\"/\"kube-root-ca.crt\""
Sep 09 21:09:54.504815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.504710    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-615c6f\"/\"default-dockercfg-gnj27\""
Sep 09 21:09:54.504815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.504709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-615c6f\"/\"sub-test-model-615c6f-kserve-self-signed-certs\""
Sep 09 21:09:54.504815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.504710    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-615c6f\"/\"openshift-service-ca.crt\""
Sep 09 21:09:54.510255 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.510234    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"]
Sep 09 21:09:54.591258 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.591221    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:09:54.608655 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.608632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tmp-dir\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.608760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.608672    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cgh4\" (UniqueName: \"kubernetes.io/projected/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-kube-api-access-9cgh4\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.608760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.608689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-dshm\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.608855 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.608774    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tls-certs\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.608855 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.608833    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-model-cache\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.608925 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.608856    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-home\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.710229 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.710195    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tmp-dir\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.710398 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.710240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cgh4\" (UniqueName: \"kubernetes.io/projected/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-kube-api-access-9cgh4\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.710398 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.710259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-dshm\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.710398 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.710294    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tls-certs\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.710398 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.710370    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-model-cache\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.710599 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.710400    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-home\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.710707 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.710689    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-home\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.710856 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.710654    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tmp-dir\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.711472 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.711442    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-model-cache\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.718826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.718401    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-dshm\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.718826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.718787    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tls-certs\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.719900 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.719878    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cgh4\" (UniqueName: \"kubernetes.io/projected/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-kube-api-access-9cgh4\") pod \"sub-test-model-615c6f-kserve-64f46d944f-sxjgq\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:54.811706 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:54.811638    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:09:55.085172 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:55.085117    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"]
Sep 09 21:09:55.087335 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:09:55.087292    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3fe0e01c_93f9_4a3d_92fe_92214f970b0f.slice/crio-fe904ccc079c9b7fb1046db0f20f0c32972f612415d3d5591a9205e64b89cae8 WatchSource:0}: Error finding container fe904ccc079c9b7fb1046db0f20f0c32972f612415d3d5591a9205e64b89cae8: Status 404 returned error can't find the container with id fe904ccc079c9b7fb1046db0f20f0c32972f612415d3d5591a9205e64b89cae8
Sep 09 21:09:55.641604 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:55.641571    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq" event={"ID":"3fe0e01c-93f9-4a3d-92fe-92214f970b0f","Type":"ContainerStarted","Data":"63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9"}
Sep 09 21:09:55.641604 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:55.641609    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq" event={"ID":"3fe0e01c-93f9-4a3d-92fe-92214f970b0f","Type":"ContainerStarted","Data":"fe904ccc079c9b7fb1046db0f20f0c32972f612415d3d5591a9205e64b89cae8"}
Sep 09 21:09:55.663241 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:55.663195    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq" podStartSLOduration=1.6631788090000001 podStartE2EDuration="1.663178809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:09:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:09:55.662291493 +0000 UTC m=+1775.854129094" watchObservedRunningTime="2026-09-09 21:09:55.663178809 +0000 UTC m=+1775.855016408"
Sep 09 21:09:57.421724 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:57.421691    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:09:59.753206 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:59.753155    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65" podUID="67a57a69-5d3e-450c-866c-d5b730edb1f0" containerName="main" containerID="cri-o://ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3" gracePeriod=45
Sep 09 21:09:59.995150 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:09:59.995125    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:10:00.055385 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055292    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcqx4\" (UniqueName: \"kubernetes.io/projected/67a57a69-5d3e-450c-866c-d5b730edb1f0-kube-api-access-rcqx4\") pod \"67a57a69-5d3e-450c-866c-d5b730edb1f0\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") "
Sep 09 21:10:00.055385 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055370    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-model-cache\") pod \"67a57a69-5d3e-450c-866c-d5b730edb1f0\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") "
Sep 09 21:10:00.055583 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055402    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a57a69-5d3e-450c-866c-d5b730edb1f0-tls-certs\") pod \"67a57a69-5d3e-450c-866c-d5b730edb1f0\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") "
Sep 09 21:10:00.055583 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055427    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-dshm\") pod \"67a57a69-5d3e-450c-866c-d5b730edb1f0\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") "
Sep 09 21:10:00.055583 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055484    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-home\") pod \"67a57a69-5d3e-450c-866c-d5b730edb1f0\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") "
Sep 09 21:10:00.055583 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055574    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-tmp-dir\") pod \"67a57a69-5d3e-450c-866c-d5b730edb1f0\" (UID: \"67a57a69-5d3e-450c-866c-d5b730edb1f0\") "
Sep 09 21:10:00.055806 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055668    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-model-cache" (OuterVolumeSpecName: "model-cache") pod "67a57a69-5d3e-450c-866c-d5b730edb1f0" (UID: "67a57a69-5d3e-450c-866c-d5b730edb1f0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:00.055864 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-home" (OuterVolumeSpecName: "home") pod "67a57a69-5d3e-450c-866c-d5b730edb1f0" (UID: "67a57a69-5d3e-450c-866c-d5b730edb1f0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:00.055930 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055856    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "67a57a69-5d3e-450c-866c-d5b730edb1f0" (UID: "67a57a69-5d3e-450c-866c-d5b730edb1f0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:00.055930 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055911    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:10:00.056055 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.055924    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:10:00.057623 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.057598    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a57a69-5d3e-450c-866c-d5b730edb1f0-kube-api-access-rcqx4" (OuterVolumeSpecName: "kube-api-access-rcqx4") pod "67a57a69-5d3e-450c-866c-d5b730edb1f0" (UID: "67a57a69-5d3e-450c-866c-d5b730edb1f0"). InnerVolumeSpecName "kube-api-access-rcqx4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:10:00.057623 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.057616    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-dshm" (OuterVolumeSpecName: "dshm") pod "67a57a69-5d3e-450c-866c-d5b730edb1f0" (UID: "67a57a69-5d3e-450c-866c-d5b730edb1f0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:10:00.057759 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.057628    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a57a69-5d3e-450c-866c-d5b730edb1f0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "67a57a69-5d3e-450c-866c-d5b730edb1f0" (UID: "67a57a69-5d3e-450c-866c-d5b730edb1f0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:10:00.156440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.156419    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:10:00.156440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.156439    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcqx4\" (UniqueName: \"kubernetes.io/projected/67a57a69-5d3e-450c-866c-d5b730edb1f0-kube-api-access-rcqx4\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:10:00.156573 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.156450    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/67a57a69-5d3e-450c-866c-d5b730edb1f0-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:10:00.156573 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.156458    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/67a57a69-5d3e-450c-866c-d5b730edb1f0-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:10:00.666142 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.666105    2566 generic.go:356] "Generic (PLEG): container finished" podID="67a57a69-5d3e-450c-866c-d5b730edb1f0" containerID="ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3" exitCode=0
Sep 09 21:10:00.666346 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.666160    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"
Sep 09 21:10:00.666346 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.666179    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65" event={"ID":"67a57a69-5d3e-450c-866c-d5b730edb1f0","Type":"ContainerDied","Data":"ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3"}
Sep 09 21:10:00.666346 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.666216    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65" event={"ID":"67a57a69-5d3e-450c-866c-d5b730edb1f0","Type":"ContainerDied","Data":"9cb4b8a09f11efaf46f998ad6ab5d82ed47f932fb5f6f63659430b5978755f6d"}
Sep 09 21:10:00.666346 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.666238    2566 scope.go:117] "RemoveContainer" containerID="ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3"
Sep 09 21:10:00.674324 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.674287    2566 scope.go:117] "RemoveContainer" containerID="ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3"
Sep 09 21:10:00.674561 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:10:00.674543    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3\": container with ID starting with ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3 not found: ID does not exist" containerID="ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3"
Sep 09 21:10:00.674622 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.674569    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3"} err="failed to get container status \"ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3\": rpc error: code = NotFound desc = could not find container \"ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3\": container with ID starting with ec55371b3b6a85403819c334cb21bed3915e248d2d80ac21516090f670bf12d3 not found: ID does not exist"
Sep 09 21:10:00.683927 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.683904    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"]
Sep 09 21:10:00.689614 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:00.689593    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/auth-test-model-0b5d4c-kserve-d5cfb48f9-4wv65"]
Sep 09 21:10:02.285299 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:02.285263    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="67a57a69-5d3e-450c-866c-d5b730edb1f0" path="/var/lib/kubelet/pods/67a57a69-5d3e-450c-866c-d5b730edb1f0/volumes"
Sep 09 21:10:04.812358 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:04.812330    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:10:04.812736 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:04.812372    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:10:04.824815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:04.824790    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:10:05.635895 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:05.635861    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:05.700851 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:05.700827    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:10:13.186496 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.185487    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"]
Sep 09 21:10:13.186496 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.186370    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67a57a69-5d3e-450c-866c-d5b730edb1f0" containerName="main"
Sep 09 21:10:13.186496 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.186399    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="67a57a69-5d3e-450c-866c-d5b730edb1f0" containerName="main"
Sep 09 21:10:13.187027 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.186588    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="67a57a69-5d3e-450c-866c-d5b730edb1f0" containerName="main"
Sep 09 21:10:13.191786 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.191763    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.194560 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.194533    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"kube-root-ca.crt\""
Sep 09 21:10:13.195282 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.195262    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"default-dockercfg-qrbw2\""
Sep 09 21:10:13.195576 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.195556    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"sub-test-model-0b5d4c-kserve-self-signed-certs\""
Sep 09 21:10:13.195661 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.195575    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"openshift-service-ca.crt\""
Sep 09 21:10:13.201695 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.201673    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"]
Sep 09 21:10:13.225914 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.225890    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:13.261568 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.261547    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56vns\" (UniqueName: \"kubernetes.io/projected/685ab26d-3348-4a4b-b968-81d0ce32a746-kube-api-access-56vns\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.261668 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.261574    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-tmp-dir\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.261668 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.261606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-home\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.261668 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.261643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-dshm\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.261821 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.261689    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685ab26d-3348-4a4b-b968-81d0ce32a746-tls-certs\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.261821 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.261758    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-model-cache\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.362789 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.362764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685ab26d-3348-4a4b-b968-81d0ce32a746-tls-certs\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.362882 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.362797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-model-cache\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.362940 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.362921    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56vns\" (UniqueName: \"kubernetes.io/projected/685ab26d-3348-4a4b-b968-81d0ce32a746-kube-api-access-56vns\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.362976 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.362961    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-tmp-dir\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.363030 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.363016    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-home\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.363067 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.363043    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-dshm\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.363127 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.363108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-model-cache\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.363355 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.363304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-tmp-dir\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.363456 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.363339    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-home\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.365096 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.365078    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-dshm\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.365291 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.365272    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685ab26d-3348-4a4b-b968-81d0ce32a746-tls-certs\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.372633 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.372612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-56vns\" (UniqueName: \"kubernetes.io/projected/685ab26d-3348-4a4b-b968-81d0ce32a746-kube-api-access-56vns\") pod \"sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.501174 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.501143    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:13.774872 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:13.774847    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"]
Sep 09 21:10:13.775887 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:10:13.775854    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod685ab26d_3348_4a4b_b968_81d0ce32a746.slice/crio-bf132f8bce5b8c50f8d734546f64c7c714e9fc30527edaa118c4306695265a56 WatchSource:0}: Error finding container bf132f8bce5b8c50f8d734546f64c7c714e9fc30527edaa118c4306695265a56: Status 404 returned error can't find the container with id bf132f8bce5b8c50f8d734546f64c7c714e9fc30527edaa118c4306695265a56
Sep 09 21:10:14.734077 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:14.734040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr" event={"ID":"685ab26d-3348-4a4b-b968-81d0ce32a746","Type":"ContainerStarted","Data":"54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb"}
Sep 09 21:10:14.734077 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:14.734084    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr" event={"ID":"685ab26d-3348-4a4b-b968-81d0ce32a746","Type":"ContainerStarted","Data":"bf132f8bce5b8c50f8d734546f64c7c714e9fc30527edaa118c4306695265a56"}
Sep 09 21:10:16.874269 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:16.874210    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr" podStartSLOduration=3.874191793 podStartE2EDuration="3.874191793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:10:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:10:14.765329037 +0000 UTC m=+1794.957166631" watchObservedRunningTime="2026-09-09 21:10:16.874191793 +0000 UTC m=+1797.066029391"
Sep 09 21:10:16.875519 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:16.875494    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:10:23.502100 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:23.502053    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:23.502504 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:23.502115    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:23.514424 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:23.514402    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:23.788278 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:23.788194    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:10:51.371649 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:51.371608    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:51.452608 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:51.452559    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:56.455485 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:56.455442    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:10:58.210196 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:10:58.210155    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:11:13.274407 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:13.274371    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:11:14.042608 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:14.042570    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"]
Sep 09 21:11:14.071019 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:14.070988    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:14.998600 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:14.998563    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"]
Sep 09 21:11:16.052645 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:16.052602    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:20.598650 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.598618    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"]
Sep 09 21:11:20.601947 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.601926    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.604846 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.604828    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-615c6f\"/\"rate-test-model-615c6f-kserve-self-signed-certs\""
Sep 09 21:11:20.612919 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.612896    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"]
Sep 09 21:11:20.619068 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.619043    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-home\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.619184 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.619080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-tmp-dir\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.619184 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.619173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tntrb\" (UniqueName: \"kubernetes.io/projected/1e18526a-6433-46fc-b22d-a20708a3c02b-kube-api-access-tntrb\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.619293 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.619208    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-dshm\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.619293 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.619245    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e18526a-6433-46fc-b22d-a20708a3c02b-tls-certs\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.619398 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.619361    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-model-cache\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.643012 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.642985    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:20.719772 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.719741    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e18526a-6433-46fc-b22d-a20708a3c02b-tls-certs\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.719899 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.719797    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-model-cache\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.719899 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.719861    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-home\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.719899 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.719884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-tmp-dir\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.720070 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.719931    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tntrb\" (UniqueName: \"kubernetes.io/projected/1e18526a-6433-46fc-b22d-a20708a3c02b-kube-api-access-tntrb\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.720124 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.720074    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-dshm\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.720222 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.720206    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-model-cache\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.720275 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.720249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-home\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.720369 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.720348    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-tmp-dir\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.722236 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.722211    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-dshm\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.722521 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.722499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e18526a-6433-46fc-b22d-a20708a3c02b-tls-certs\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.728596 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.728573    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tntrb\" (UniqueName: \"kubernetes.io/projected/1e18526a-6433-46fc-b22d-a20708a3c02b-kube-api-access-tntrb\") pod \"rate-test-model-615c6f-kserve-68dd6bcd87-7zghj\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") " pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:20.911517 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:20.911441    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:21.212034 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:21.211906    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"]
Sep 09 21:11:21.215054 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:11:21.215026    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e18526a_6433_46fc_b22d_a20708a3c02b.slice/crio-511356117b9ec09d0c810322d5a961be5f51a240af427c3ddc66ecf70f443610 WatchSource:0}: Error finding container 511356117b9ec09d0c810322d5a961be5f51a240af427c3ddc66ecf70f443610: Status 404 returned error can't find the container with id 511356117b9ec09d0c810322d5a961be5f51a240af427c3ddc66ecf70f443610
Sep 09 21:11:22.049072 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:22.049036    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj" event={"ID":"1e18526a-6433-46fc-b22d-a20708a3c02b","Type":"ContainerStarted","Data":"099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe"}
Sep 09 21:11:22.049502 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:22.049079    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj" event={"ID":"1e18526a-6433-46fc-b22d-a20708a3c02b","Type":"ContainerStarted","Data":"511356117b9ec09d0c810322d5a961be5f51a240af427c3ddc66ecf70f443610"}
Sep 09 21:11:22.068676 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:22.068617    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj" podStartSLOduration=2.068598107 podStartE2EDuration="2.068598107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:22.067410254 +0000 UTC m=+1862.259247853" watchObservedRunningTime="2026-09-09 21:11:22.068598107 +0000 UTC m=+1862.260435712"
Sep 09 21:11:23.377760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:23.377725    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:11:24.869248 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:24.869206    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:29.061830 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.061777    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq" podUID="3fe0e01c-93f9-4a3d-92fe-92214f970b0f" containerName="main" containerID="cri-o://63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9" gracePeriod=45
Sep 09 21:11:29.313323 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.313254    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:11:29.392355 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392301    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-home\") pod \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") "
Sep 09 21:11:29.392525 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392364    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tls-certs\") pod \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") "
Sep 09 21:11:29.392525 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392411    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tmp-dir\") pod \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") "
Sep 09 21:11:29.392525 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392473    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cgh4\" (UniqueName: \"kubernetes.io/projected/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-kube-api-access-9cgh4\") pod \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") "
Sep 09 21:11:29.392525 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392499    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-model-cache\") pod \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") "
Sep 09 21:11:29.392742 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392561    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-dshm\") pod \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\" (UID: \"3fe0e01c-93f9-4a3d-92fe-92214f970b0f\") "
Sep 09 21:11:29.392742 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392591    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-home" (OuterVolumeSpecName: "home") pod "3fe0e01c-93f9-4a3d-92fe-92214f970b0f" (UID: "3fe0e01c-93f9-4a3d-92fe-92214f970b0f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:29.392846 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392730    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3fe0e01c-93f9-4a3d-92fe-92214f970b0f" (UID: "3fe0e01c-93f9-4a3d-92fe-92214f970b0f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:29.392846 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392805    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-model-cache" (OuterVolumeSpecName: "model-cache") pod "3fe0e01c-93f9-4a3d-92fe-92214f970b0f" (UID: "3fe0e01c-93f9-4a3d-92fe-92214f970b0f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:29.392914 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392854    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:29.392914 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.392874    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:29.394724 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.394695    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-dshm" (OuterVolumeSpecName: "dshm") pod "3fe0e01c-93f9-4a3d-92fe-92214f970b0f" (UID: "3fe0e01c-93f9-4a3d-92fe-92214f970b0f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:29.394724 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.394710    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3fe0e01c-93f9-4a3d-92fe-92214f970b0f" (UID: "3fe0e01c-93f9-4a3d-92fe-92214f970b0f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:11:29.394865 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.394758    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-kube-api-access-9cgh4" (OuterVolumeSpecName: "kube-api-access-9cgh4") pod "3fe0e01c-93f9-4a3d-92fe-92214f970b0f" (UID: "3fe0e01c-93f9-4a3d-92fe-92214f970b0f"). InnerVolumeSpecName "kube-api-access-9cgh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:11:29.493985 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.493942    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:29.493985 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.493978    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:29.493985 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.493990    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cgh4\" (UniqueName: \"kubernetes.io/projected/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-kube-api-access-9cgh4\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:29.494223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:29.493999    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3fe0e01c-93f9-4a3d-92fe-92214f970b0f-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:30.018433 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.018388    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr" podUID="685ab26d-3348-4a4b-b968-81d0ce32a746" containerName="main" containerID="cri-o://54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb" gracePeriod=45
Sep 09 21:11:30.087596 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.087561    2566 generic.go:356] "Generic (PLEG): container finished" podID="3fe0e01c-93f9-4a3d-92fe-92214f970b0f" containerID="63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9" exitCode=0
Sep 09 21:11:30.087945 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.087592    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq" event={"ID":"3fe0e01c-93f9-4a3d-92fe-92214f970b0f","Type":"ContainerDied","Data":"63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9"}
Sep 09 21:11:30.087945 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.087603    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"
Sep 09 21:11:30.087945 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.087637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq" event={"ID":"3fe0e01c-93f9-4a3d-92fe-92214f970b0f","Type":"ContainerDied","Data":"fe904ccc079c9b7fb1046db0f20f0c32972f612415d3d5591a9205e64b89cae8"}
Sep 09 21:11:30.087945 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.087655    2566 scope.go:117] "RemoveContainer" containerID="63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9"
Sep 09 21:11:30.108146 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.108120    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"]
Sep 09 21:11:30.114413 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.114392    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/sub-test-model-615c6f-kserve-64f46d944f-sxjgq"]
Sep 09 21:11:30.129361 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.129337    2566 scope.go:117] "RemoveContainer" containerID="63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9"
Sep 09 21:11:30.129761 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:11:30.129738    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9\": container with ID starting with 63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9 not found: ID does not exist" containerID="63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9"
Sep 09 21:11:30.129861 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.129769    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9"} err="failed to get container status \"63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9\": rpc error: code = NotFound desc = could not find container \"63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9\": container with ID starting with 63b4ed3626ee164d22743eb595807a140c225081166e1a123170103a64859ce9 not found: ID does not exist"
Sep 09 21:11:30.285462 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.285437    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:11:30.285690 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.285670    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3fe0e01c-93f9-4a3d-92fe-92214f970b0f" path="/var/lib/kubelet/pods/3fe0e01c-93f9-4a3d-92fe-92214f970b0f/volumes"
Sep 09 21:11:30.301299 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301275    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-model-cache\") pod \"685ab26d-3348-4a4b-b968-81d0ce32a746\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") "
Sep 09 21:11:30.301421 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301347    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-dshm\") pod \"685ab26d-3348-4a4b-b968-81d0ce32a746\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") "
Sep 09 21:11:30.301421 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301373    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-home\") pod \"685ab26d-3348-4a4b-b968-81d0ce32a746\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") "
Sep 09 21:11:30.301421 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301391    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-tmp-dir\") pod \"685ab26d-3348-4a4b-b968-81d0ce32a746\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") "
Sep 09 21:11:30.301421 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301418    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685ab26d-3348-4a4b-b968-81d0ce32a746-tls-certs\") pod \"685ab26d-3348-4a4b-b968-81d0ce32a746\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") "
Sep 09 21:11:30.301628 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301463    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-56vns\" (UniqueName: \"kubernetes.io/projected/685ab26d-3348-4a4b-b968-81d0ce32a746-kube-api-access-56vns\") pod \"685ab26d-3348-4a4b-b968-81d0ce32a746\" (UID: \"685ab26d-3348-4a4b-b968-81d0ce32a746\") "
Sep 09 21:11:30.301628 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301502    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-model-cache" (OuterVolumeSpecName: "model-cache") pod "685ab26d-3348-4a4b-b968-81d0ce32a746" (UID: "685ab26d-3348-4a4b-b968-81d0ce32a746"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:30.301749 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301645    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-home" (OuterVolumeSpecName: "home") pod "685ab26d-3348-4a4b-b968-81d0ce32a746" (UID: "685ab26d-3348-4a4b-b968-81d0ce32a746"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:30.301749 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "685ab26d-3348-4a4b-b968-81d0ce32a746" (UID: "685ab26d-3348-4a4b-b968-81d0ce32a746"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:30.301852 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301808    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:30.301852 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301827    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:30.301852 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.301842    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:30.303775 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.303751    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/685ab26d-3348-4a4b-b968-81d0ce32a746-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "685ab26d-3348-4a4b-b968-81d0ce32a746" (UID: "685ab26d-3348-4a4b-b968-81d0ce32a746"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:11:30.303895 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.303871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/685ab26d-3348-4a4b-b968-81d0ce32a746-kube-api-access-56vns" (OuterVolumeSpecName: "kube-api-access-56vns") pod "685ab26d-3348-4a4b-b968-81d0ce32a746" (UID: "685ab26d-3348-4a4b-b968-81d0ce32a746"). InnerVolumeSpecName "kube-api-access-56vns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:11:30.304050 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.303996    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-dshm" (OuterVolumeSpecName: "dshm") pod "685ab26d-3348-4a4b-b968-81d0ce32a746" (UID: "685ab26d-3348-4a4b-b968-81d0ce32a746"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:11:30.402715 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.402674    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56vns\" (UniqueName: \"kubernetes.io/projected/685ab26d-3348-4a4b-b968-81d0ce32a746-kube-api-access-56vns\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:30.402715 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.402713    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/685ab26d-3348-4a4b-b968-81d0ce32a746-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:30.402715 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.402725    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/685ab26d-3348-4a4b-b968-81d0ce32a746-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:11:30.911917 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.911881    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:30.912096 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.911934    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:30.923883 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:30.923859    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:31.093930 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.093901    2566 generic.go:356] "Generic (PLEG): container finished" podID="685ab26d-3348-4a4b-b968-81d0ce32a746" containerID="54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb" exitCode=0
Sep 09 21:11:31.094393 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.093946    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"
Sep 09 21:11:31.094393 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.093987    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr" event={"ID":"685ab26d-3348-4a4b-b968-81d0ce32a746","Type":"ContainerDied","Data":"54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb"}
Sep 09 21:11:31.094393 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.094040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr" event={"ID":"685ab26d-3348-4a4b-b968-81d0ce32a746","Type":"ContainerDied","Data":"bf132f8bce5b8c50f8d734546f64c7c714e9fc30527edaa118c4306695265a56"}
Sep 09 21:11:31.094393 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.094067    2566 scope.go:117] "RemoveContainer" containerID="54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb"
Sep 09 21:11:31.103283 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.103264    2566 scope.go:117] "RemoveContainer" containerID="54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb"
Sep 09 21:11:31.103572 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:11:31.103553    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb\": container with ID starting with 54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb not found: ID does not exist" containerID="54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb"
Sep 09 21:11:31.103633 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.103581    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb"} err="failed to get container status \"54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb\": rpc error: code = NotFound desc = could not find container \"54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb\": container with ID starting with 54617e5a7486362329c6731c1fde1079e96604f00fa5348703c68062ae7cb0eb not found: ID does not exist"
Sep 09 21:11:31.107210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.107184    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:11:31.118508 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.118486    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"]
Sep 09 21:11:31.125096 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:31.123111    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/sub-test-model-0b5d4c-kserve-6b8c4f5995-hdcsr"]
Sep 09 21:11:32.285879 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:32.285846    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="685ab26d-3348-4a4b-b968-81d0ce32a746" path="/var/lib/kubelet/pods/685ab26d-3348-4a4b-b968-81d0ce32a746/volumes"
Sep 09 21:11:44.968698 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.968620    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"]
Sep 09 21:11:44.969112 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.968973    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="685ab26d-3348-4a4b-b968-81d0ce32a746" containerName="main"
Sep 09 21:11:44.969112 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.968984    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="685ab26d-3348-4a4b-b968-81d0ce32a746" containerName="main"
Sep 09 21:11:44.969112 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.969001    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3fe0e01c-93f9-4a3d-92fe-92214f970b0f" containerName="main"
Sep 09 21:11:44.969112 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.969007    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="3fe0e01c-93f9-4a3d-92fe-92214f970b0f" containerName="main"
Sep 09 21:11:44.969112 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.969065    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="685ab26d-3348-4a4b-b968-81d0ce32a746" containerName="main"
Sep 09 21:11:44.969112 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.969077    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="3fe0e01c-93f9-4a3d-92fe-92214f970b0f" containerName="main"
Sep 09 21:11:44.972986 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.972965    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:44.975607 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.975586    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"kube-root-ca.crt\""
Sep 09 21:11:44.976585 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.976564    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"openshift-service-ca.crt\""
Sep 09 21:11:44.976689 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.976570    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"rate-test-model-0b5d4c-kserve-self-signed-certs\""
Sep 09 21:11:44.976770 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.976569    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-0b5d4c\"/\"default-dockercfg-qrbw2\""
Sep 09 21:11:44.990760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.990742    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"]
Sep 09 21:11:44.997098 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:44.997070    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:45.025972 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.025948    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-dshm\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.026081 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.025981    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-home\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.026081 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.026005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-model-cache\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.026155 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.026075    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwwmq\" (UniqueName: \"kubernetes.io/projected/ae03d410-131d-4162-8537-10f57870179b-kube-api-access-nwwmq\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.026155 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.026098    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-tmp-dir\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.026155 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.026134    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae03d410-131d-4162-8537-10f57870179b-tls-certs\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.126744 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.126712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae03d410-131d-4162-8537-10f57870179b-tls-certs\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.126908 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.126868    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-dshm\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.126908 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.126891    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-home\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.127063 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.126908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-model-cache\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.127063 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.126928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwwmq\" (UniqueName: \"kubernetes.io/projected/ae03d410-131d-4162-8537-10f57870179b-kube-api-access-nwwmq\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.127063 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.126947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-tmp-dir\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.127349 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.127320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-home\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.127476 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.127364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-model-cache\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.127476 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.127398    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-tmp-dir\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.129109 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.129090    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-dshm\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.129284 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.129266    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae03d410-131d-4162-8537-10f57870179b-tls-certs\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.135894 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.135871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwwmq\" (UniqueName: \"kubernetes.io/projected/ae03d410-131d-4162-8537-10f57870179b-kube-api-access-nwwmq\") pod \"rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") " pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.283018 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.282986    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:45.555817 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:45.555792    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"]
Sep 09 21:11:45.558126 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:11:45.558098    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae03d410_131d_4162_8537_10f57870179b.slice/crio-069e8447f744e8dcebe37368706c1edc76120ff9e68eca64281224253767627d WatchSource:0}: Error finding container 069e8447f744e8dcebe37368706c1edc76120ff9e68eca64281224253767627d: Status 404 returned error can't find the container with id 069e8447f744e8dcebe37368706c1edc76120ff9e68eca64281224253767627d
Sep 09 21:11:46.169037 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:46.169005    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l" event={"ID":"ae03d410-131d-4162-8537-10f57870179b","Type":"ContainerStarted","Data":"24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f"}
Sep 09 21:11:46.169421 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:46.169045    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l" event={"ID":"ae03d410-131d-4162-8537-10f57870179b","Type":"ContainerStarted","Data":"069e8447f744e8dcebe37368706c1edc76120ff9e68eca64281224253767627d"}
Sep 09 21:11:46.222486 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:46.222428    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l" podStartSLOduration=2.222410275 podStartE2EDuration="2.222410275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:11:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:11:46.22147986 +0000 UTC m=+1886.413317530" watchObservedRunningTime="2026-09-09 21:11:46.222410275 +0000 UTC m=+1886.414247876"
Sep 09 21:11:48.380767 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:48.380732    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:11:54.807599 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:54.807565    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:11:55.283502 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:55.283472    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:55.283502 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:55.283513    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:55.296163 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:55.296133    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:11:56.228217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:11:56.228164    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:12:03.860515 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:03.860479    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:12:13.181833 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:13.181793    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:12:18.748009 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:18.747974    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"]
Sep 09 21:12:18.782285 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:18.782248    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:19.731139 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:19.731100    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"]
Sep 09 21:12:19.777611 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:19.777575    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:20.155497 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:20.155413    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:20.492620 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:20.492582    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:20.764709 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:20.764586    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:23.725331 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:23.725257    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:23.820094 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:23.820047    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:24.902192 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:24.902144    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:24.991128 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:24.991093    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:25.089406 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.089373    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:25.440462 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.440421    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:25.555728 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.555689    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:25.608531 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.608502    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"]
Sep 09 21:12:25.616386 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.616363    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:25.620413 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.620389    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7c0011-dockercfg-74rwp\""
Sep 09 21:12:25.620573 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.620518    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7c0011\""
Sep 09 21:12:25.629823 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.629797    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"]
Sep 09 21:12:25.691177 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.691095    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af48d12-1074-456f-a35a-61037f0ca438-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl\" (UID: \"0af48d12-1074-456f-a35a-61037f0ca438\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:25.691177 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.691133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn6vf\" (UniqueName: \"kubernetes.io/projected/0af48d12-1074-456f-a35a-61037f0ca438-kube-api-access-pn6vf\") pod \"payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl\" (UID: \"0af48d12-1074-456f-a35a-61037f0ca438\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:25.792566 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.792536    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af48d12-1074-456f-a35a-61037f0ca438-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl\" (UID: \"0af48d12-1074-456f-a35a-61037f0ca438\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:25.792720 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.792572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn6vf\" (UniqueName: \"kubernetes.io/projected/0af48d12-1074-456f-a35a-61037f0ca438-kube-api-access-pn6vf\") pod \"payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl\" (UID: \"0af48d12-1074-456f-a35a-61037f0ca438\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:25.793241 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.793222    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af48d12-1074-456f-a35a-61037f0ca438-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl\" (UID: \"0af48d12-1074-456f-a35a-61037f0ca438\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:25.802067 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.802042    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn6vf\" (UniqueName: \"kubernetes.io/projected/0af48d12-1074-456f-a35a-61037f0ca438-kube-api-access-pn6vf\") pod \"payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl\" (UID: \"0af48d12-1074-456f-a35a-61037f0ca438\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:25.928049 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:25.928017    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:26.207735 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:26.207708    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"]
Sep 09 21:12:26.209924 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:12:26.209897    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af48d12_1074_456f_a35a_61037f0ca438.slice/crio-edee51682709a6778ca529a491d929db2868067633020bc1d570c1f3d92d8bea WatchSource:0}: Error finding container edee51682709a6778ca529a491d929db2868067633020bc1d570c1f3d92d8bea: Status 404 returned error can't find the container with id edee51682709a6778ca529a491d929db2868067633020bc1d570c1f3d92d8bea
Sep 09 21:12:26.359943 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:26.359911    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl" event={"ID":"0af48d12-1074-456f-a35a-61037f0ca438","Type":"ContainerStarted","Data":"b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd"}
Sep 09 21:12:26.360131 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:26.359967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl" event={"ID":"0af48d12-1074-456f-a35a-61037f0ca438","Type":"ContainerStarted","Data":"edee51682709a6778ca529a491d929db2868067633020bc1d570c1f3d92d8bea"}
Sep 09 21:12:26.360131 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:26.360014    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:26.381582 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:26.381538    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl" podStartSLOduration=1.381524295 podStartE2EDuration="1.381524295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:12:26.380521192 +0000 UTC m=+1926.572358790" watchObservedRunningTime="2026-09-09 21:12:26.381524295 +0000 UTC m=+1926.573361894"
Sep 09 21:12:29.268100 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:29.268062    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:12:33.766929 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:33.766888    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj" podUID="1e18526a-6433-46fc-b22d-a20708a3c02b" containerName="main" containerID="cri-o://099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe" gracePeriod=45
Sep 09 21:12:34.005147 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.005121    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:12:34.069216 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069142    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-model-cache\") pod \"1e18526a-6433-46fc-b22d-a20708a3c02b\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") "
Sep 09 21:12:34.069216 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069192    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-tmp-dir\") pod \"1e18526a-6433-46fc-b22d-a20708a3c02b\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") "
Sep 09 21:12:34.069453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069252    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e18526a-6433-46fc-b22d-a20708a3c02b-tls-certs\") pod \"1e18526a-6433-46fc-b22d-a20708a3c02b\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") "
Sep 09 21:12:34.069453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069289    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tntrb\" (UniqueName: \"kubernetes.io/projected/1e18526a-6433-46fc-b22d-a20708a3c02b-kube-api-access-tntrb\") pod \"1e18526a-6433-46fc-b22d-a20708a3c02b\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") "
Sep 09 21:12:34.069453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069377    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-dshm\") pod \"1e18526a-6433-46fc-b22d-a20708a3c02b\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") "
Sep 09 21:12:34.069453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069417    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-home\") pod \"1e18526a-6433-46fc-b22d-a20708a3c02b\" (UID: \"1e18526a-6433-46fc-b22d-a20708a3c02b\") "
Sep 09 21:12:34.069658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069450    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1e18526a-6433-46fc-b22d-a20708a3c02b" (UID: "1e18526a-6433-46fc-b22d-a20708a3c02b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:34.069658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069533    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1e18526a-6433-46fc-b22d-a20708a3c02b" (UID: "1e18526a-6433-46fc-b22d-a20708a3c02b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:34.069770 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069694    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-home" (OuterVolumeSpecName: "home") pod "1e18526a-6433-46fc-b22d-a20708a3c02b" (UID: "1e18526a-6433-46fc-b22d-a20708a3c02b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:34.069833 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069806    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:34.069833 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069825    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:34.069919 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.069837    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:34.071351 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.071326    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e18526a-6433-46fc-b22d-a20708a3c02b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1e18526a-6433-46fc-b22d-a20708a3c02b" (UID: "1e18526a-6433-46fc-b22d-a20708a3c02b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:34.071440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.071397    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-dshm" (OuterVolumeSpecName: "dshm") pod "1e18526a-6433-46fc-b22d-a20708a3c02b" (UID: "1e18526a-6433-46fc-b22d-a20708a3c02b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:34.071440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.071411    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e18526a-6433-46fc-b22d-a20708a3c02b-kube-api-access-tntrb" (OuterVolumeSpecName: "kube-api-access-tntrb") pod "1e18526a-6433-46fc-b22d-a20708a3c02b" (UID: "1e18526a-6433-46fc-b22d-a20708a3c02b"). InnerVolumeSpecName "kube-api-access-tntrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:34.170238 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.170216    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e18526a-6433-46fc-b22d-a20708a3c02b-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:34.170238 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.170237    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tntrb\" (UniqueName: \"kubernetes.io/projected/1e18526a-6433-46fc-b22d-a20708a3c02b-kube-api-access-tntrb\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:34.170409 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.170246    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e18526a-6433-46fc-b22d-a20708a3c02b-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:34.399411 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.399322    2566 generic.go:356] "Generic (PLEG): container finished" podID="1e18526a-6433-46fc-b22d-a20708a3c02b" containerID="099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe" exitCode=0
Sep 09 21:12:34.399411 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.399371    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"
Sep 09 21:12:34.399411 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.399399    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj" event={"ID":"1e18526a-6433-46fc-b22d-a20708a3c02b","Type":"ContainerDied","Data":"099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe"}
Sep 09 21:12:34.399633 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.399447    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj" event={"ID":"1e18526a-6433-46fc-b22d-a20708a3c02b","Type":"ContainerDied","Data":"511356117b9ec09d0c810322d5a961be5f51a240af427c3ddc66ecf70f443610"}
Sep 09 21:12:34.399633 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.399464    2566 scope.go:117] "RemoveContainer" containerID="099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe"
Sep 09 21:12:34.407651 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.407632    2566 scope.go:117] "RemoveContainer" containerID="099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe"
Sep 09 21:12:34.407914 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:12:34.407896    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe\": container with ID starting with 099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe not found: ID does not exist" containerID="099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe"
Sep 09 21:12:34.407966 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.407922    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe"} err="failed to get container status \"099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe\": rpc error: code = NotFound desc = could not find container \"099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe\": container with ID starting with 099c0d987cee3df970c60258f9dea17cd172c0dd8370c68a458fc19a185cfdbe not found: ID does not exist"
Sep 09 21:12:34.416854 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.416832    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"]
Sep 09 21:12:34.420363 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.420342    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-615c6f/rate-test-model-615c6f-kserve-68dd6bcd87-7zghj"]
Sep 09 21:12:34.755567 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:34.755530    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l" podUID="ae03d410-131d-4162-8537-10f57870179b" containerName="main" containerID="cri-o://24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f" gracePeriod=45
Sep 09 21:12:35.005751 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.005697    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:12:35.077687 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.077659    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-tmp-dir\") pod \"ae03d410-131d-4162-8537-10f57870179b\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") "
Sep 09 21:12:35.077837 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.077702    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-dshm\") pod \"ae03d410-131d-4162-8537-10f57870179b\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") "
Sep 09 21:12:35.077837 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.077721    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwwmq\" (UniqueName: \"kubernetes.io/projected/ae03d410-131d-4162-8537-10f57870179b-kube-api-access-nwwmq\") pod \"ae03d410-131d-4162-8537-10f57870179b\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") "
Sep 09 21:12:35.077837 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.077744    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-home\") pod \"ae03d410-131d-4162-8537-10f57870179b\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") "
Sep 09 21:12:35.077837 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.077779    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-model-cache\") pod \"ae03d410-131d-4162-8537-10f57870179b\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") "
Sep 09 21:12:35.077837 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.077828    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae03d410-131d-4162-8537-10f57870179b-tls-certs\") pod \"ae03d410-131d-4162-8537-10f57870179b\" (UID: \"ae03d410-131d-4162-8537-10f57870179b\") "
Sep 09 21:12:35.078120 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.077915    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ae03d410-131d-4162-8537-10f57870179b" (UID: "ae03d410-131d-4162-8537-10f57870179b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:35.078120 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.078031    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-home" (OuterVolumeSpecName: "home") pod "ae03d410-131d-4162-8537-10f57870179b" (UID: "ae03d410-131d-4162-8537-10f57870179b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:35.078120 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.078092    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-model-cache" (OuterVolumeSpecName: "model-cache") pod "ae03d410-131d-4162-8537-10f57870179b" (UID: "ae03d410-131d-4162-8537-10f57870179b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:35.078239 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.078186    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:35.078239 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.078207    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:35.078239 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.078221    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:35.079831 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.079801    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-dshm" (OuterVolumeSpecName: "dshm") pod "ae03d410-131d-4162-8537-10f57870179b" (UID: "ae03d410-131d-4162-8537-10f57870179b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:12:35.079942 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.079854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae03d410-131d-4162-8537-10f57870179b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ae03d410-131d-4162-8537-10f57870179b" (UID: "ae03d410-131d-4162-8537-10f57870179b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:12:35.079942 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.079862    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae03d410-131d-4162-8537-10f57870179b-kube-api-access-nwwmq" (OuterVolumeSpecName: "kube-api-access-nwwmq") pod "ae03d410-131d-4162-8537-10f57870179b" (UID: "ae03d410-131d-4162-8537-10f57870179b"). InnerVolumeSpecName "kube-api-access-nwwmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:12:35.179226 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.179202    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ae03d410-131d-4162-8537-10f57870179b-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:35.179226 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.179223    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwwmq\" (UniqueName: \"kubernetes.io/projected/ae03d410-131d-4162-8537-10f57870179b-kube-api-access-nwwmq\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:35.179403 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.179233    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ae03d410-131d-4162-8537-10f57870179b-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:12:35.411594 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.411508    2566 generic.go:356] "Generic (PLEG): container finished" podID="ae03d410-131d-4162-8537-10f57870179b" containerID="24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f" exitCode=0
Sep 09 21:12:35.411594 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.411555    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"
Sep 09 21:12:35.411827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.411600    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l" event={"ID":"ae03d410-131d-4162-8537-10f57870179b","Type":"ContainerDied","Data":"24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f"}
Sep 09 21:12:35.411827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.411648    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l" event={"ID":"ae03d410-131d-4162-8537-10f57870179b","Type":"ContainerDied","Data":"069e8447f744e8dcebe37368706c1edc76120ff9e68eca64281224253767627d"}
Sep 09 21:12:35.411827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.411669    2566 scope.go:117] "RemoveContainer" containerID="24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f"
Sep 09 21:12:35.420884 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.420870    2566 scope.go:117] "RemoveContainer" containerID="24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f"
Sep 09 21:12:35.421119 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:12:35.421101    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f\": container with ID starting with 24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f not found: ID does not exist" containerID="24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f"
Sep 09 21:12:35.421196 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.421128    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f"} err="failed to get container status \"24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f\": rpc error: code = NotFound desc = could not find container \"24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f\": container with ID starting with 24c050c6008400bdcbd775dd3b8b796d305c1960923f9cd13453087f443de10f not found: ID does not exist"
Sep 09 21:12:35.432889 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.432868    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"]
Sep 09 21:12:35.435236 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:35.435215    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-0b5d4c/rate-test-model-0b5d4c-kserve-f45b7584c-lcb2l"]
Sep 09 21:12:36.286294 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:36.286253    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e18526a-6433-46fc-b22d-a20708a3c02b" path="/var/lib/kubelet/pods/1e18526a-6433-46fc-b22d-a20708a3c02b/volumes"
Sep 09 21:12:36.286826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:36.286806    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae03d410-131d-4162-8537-10f57870179b" path="/var/lib/kubelet/pods/ae03d410-131d-4162-8537-10f57870179b/volumes"
Sep 09 21:12:37.367408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:37.367376    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:12:48.624408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:48.624369    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:49.297556 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.297526    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:49.603130 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.603049    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"]
Sep 09 21:12:49.603712 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.603569    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae03d410-131d-4162-8537-10f57870179b" containerName="main"
Sep 09 21:12:49.603712 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.603590    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae03d410-131d-4162-8537-10f57870179b" containerName="main"
Sep 09 21:12:49.603712 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.603618    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e18526a-6433-46fc-b22d-a20708a3c02b" containerName="main"
Sep 09 21:12:49.603712 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.603627    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e18526a-6433-46fc-b22d-a20708a3c02b" containerName="main"
Sep 09 21:12:49.603938 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.603721    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae03d410-131d-4162-8537-10f57870179b" containerName="main"
Sep 09 21:12:49.603938 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.603737    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e18526a-6433-46fc-b22d-a20708a3c02b" containerName="main"
Sep 09 21:12:49.606803 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.606783    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.610914 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.610886    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-7d51a3-openshift-default-dockercfg-m4576\""
Sep 09 21:12:49.627553 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.627531    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"]
Sep 09 21:12:49.649595 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.649567    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:49.702118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.702090    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istiod-ca-cert\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.702277 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.702137    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-credential-socket\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.702277 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.702178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-data\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.702277 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.702205    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-token\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.702487 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.702363    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-socket\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.702487 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.702391    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hxxd\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-kube-api-access-4hxxd\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.702487 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.702425    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-podinfo\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.702487 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.702450    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-envoy\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.702708 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.702546    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-certs\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.803843 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.803813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-podinfo\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.803991 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.803845    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-envoy\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.803991 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.803879    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-certs\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.803991 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.803904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istiod-ca-cert\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.803991 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.803933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-credential-socket\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.803991 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.803972    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-data\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.804267 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.804005    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-token\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.804267 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.804067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-socket\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.804267 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.804091    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hxxd\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-kube-api-access-4hxxd\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.804468 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.804444    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-data\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.804666 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.804642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-socket\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.804725 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.804660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-certs\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.804760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.804744    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istiod-ca-cert\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.804796 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.804762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-credential-socket\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.806207 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.806188    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-envoy\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.806373 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.806354    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-podinfo\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.812784 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.812759    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-token\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.814938 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.814262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hxxd\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-kube-api-access-4hxxd\") pod \"e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") " pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.916665 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.916578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:49.953934 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:49.953899    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:50.199927 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:50.199899    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"]
Sep 09 21:12:50.202142 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:12:50.202116    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebc421cd_26fa_41e4_81f1_f8783acb28e1.slice/crio-7f79a9bab2362b3c6e27bd9adf101d1240ea248ab232525d8c63ba6d02c80e31 WatchSource:0}: Error finding container 7f79a9bab2362b3c6e27bd9adf101d1240ea248ab232525d8c63ba6d02c80e31: Status 404 returned error can't find the container with id 7f79a9bab2362b3c6e27bd9adf101d1240ea248ab232525d8c63ba6d02c80e31
Sep 09 21:12:50.204571 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:50.204537    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:12:50.204682 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:50.204614    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:12:50.204682 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:50.204658    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:12:50.489151 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:50.489115    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" event={"ID":"ebc421cd-26fa-41e4-81f1-f8783acb28e1","Type":"ContainerStarted","Data":"ff9b50e57307a21568a036f8fc37bf05c66aee29be4aa0ffb875e6e7fbd78fbd"}
Sep 09 21:12:50.489352 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:50.489170    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" event={"ID":"ebc421cd-26fa-41e4-81f1-f8783acb28e1","Type":"ContainerStarted","Data":"7f79a9bab2362b3c6e27bd9adf101d1240ea248ab232525d8c63ba6d02c80e31"}
Sep 09 21:12:50.513194 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:50.513127    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" podStartSLOduration=1.513106448 podStartE2EDuration="1.513106448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:12:50.51061043 +0000 UTC m=+1950.702448029" watchObservedRunningTime="2026-09-09 21:12:50.513106448 +0000 UTC m=+1950.704944048"
Sep 09 21:12:50.916827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:50.916749    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:51.922595 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:51.922566    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:52.499805 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:52.499773    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:52.500827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:52.500804    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:12:52.594440 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:52.594407    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:52.701858 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:52.701825    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:53.471490 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:53.471456    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:53.756042 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:53.756011    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:53.890090 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:53.890052    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:54.139937 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.139850    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:54.376611 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.376576    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:54.476807 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.476722    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:12:54.501883 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.501847    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"]
Sep 09 21:12:54.511542 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.511515    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:12:54.515419 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.515398    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-7d51a3\""
Sep 09 21:12:54.521513 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.521487    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"]
Sep 09 21:12:54.644843 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.644799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03820efb-7921-4926-8b09-2a62d1c2368e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7\" (UID: \"03820efb-7921-4926-8b09-2a62d1c2368e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:12:54.645039 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.644917    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6882\" (UniqueName: \"kubernetes.io/projected/03820efb-7921-4926-8b09-2a62d1c2368e-kube-api-access-k6882\") pod \"payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7\" (UID: \"03820efb-7921-4926-8b09-2a62d1c2368e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:12:54.746317 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.746279    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03820efb-7921-4926-8b09-2a62d1c2368e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7\" (UID: \"03820efb-7921-4926-8b09-2a62d1c2368e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:12:54.746488 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.746378    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6882\" (UniqueName: \"kubernetes.io/projected/03820efb-7921-4926-8b09-2a62d1c2368e-kube-api-access-k6882\") pod \"payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7\" (UID: \"03820efb-7921-4926-8b09-2a62d1c2368e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:12:54.747027 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.747007    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03820efb-7921-4926-8b09-2a62d1c2368e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7\" (UID: \"03820efb-7921-4926-8b09-2a62d1c2368e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:12:54.755279 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.755258    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6882\" (UniqueName: \"kubernetes.io/projected/03820efb-7921-4926-8b09-2a62d1c2368e-kube-api-access-k6882\") pod \"payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7\" (UID: \"03820efb-7921-4926-8b09-2a62d1c2368e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:12:54.823062 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:54.823019    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:12:55.118479 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:55.118437    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"]
Sep 09 21:12:55.121284 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:12:55.121259    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03820efb_7921_4926_8b09_2a62d1c2368e.slice/crio-3a566af8725e70356485b8a32771d1619f9eac1a97fa70d90c872e817c257d8a WatchSource:0}: Error finding container 3a566af8725e70356485b8a32771d1619f9eac1a97fa70d90c872e817c257d8a: Status 404 returned error can't find the container with id 3a566af8725e70356485b8a32771d1619f9eac1a97fa70d90c872e817c257d8a
Sep 09 21:12:55.520275 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:55.520243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7" event={"ID":"03820efb-7921-4926-8b09-2a62d1c2368e","Type":"ContainerStarted","Data":"1fb38c31e8e49b37c94cf86bb6cffae8dd4726505dbd6c0b4eb2400c2ed7fb4f"}
Sep 09 21:12:55.520760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:55.520294    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7" event={"ID":"03820efb-7921-4926-8b09-2a62d1c2368e","Type":"ContainerStarted","Data":"3a566af8725e70356485b8a32771d1619f9eac1a97fa70d90c872e817c257d8a"}
Sep 09 21:12:55.520760 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:55.520337    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:12:59.785727 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:59.785660    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7" podStartSLOduration=5.785637628 podStartE2EDuration="5.785637628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:12:55.544131292 +0000 UTC m=+1955.735968890" watchObservedRunningTime="2026-09-09 21:12:59.785637628 +0000 UTC m=+1959.977475228"
Sep 09 21:12:59.787634 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:12:59.787612    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:13:06.527833 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:06.527798    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:13:08.542464 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:08.542427    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 09 21:13:08.542799 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:08.542478    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 09 21:13:09.592442 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:09.592413    2566 generic.go:356] "Generic (PLEG): container finished" podID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerID="ff9b50e57307a21568a036f8fc37bf05c66aee29be4aa0ffb875e6e7fbd78fbd" exitCode=0
Sep 09 21:13:09.592823 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:09.592485    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" event={"ID":"ebc421cd-26fa-41e4-81f1-f8783acb28e1","Type":"ContainerDied","Data":"ff9b50e57307a21568a036f8fc37bf05c66aee29be4aa0ffb875e6e7fbd78fbd"}
Sep 09 21:13:09.592897 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:09.592883    2566 scope.go:117] "RemoveContainer" containerID="ff9b50e57307a21568a036f8fc37bf05c66aee29be4aa0ffb875e6e7fbd78fbd"
Sep 09 21:13:09.593975 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:09.593951    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:13:09.594047 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:09.594014    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:13:09.594047 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:09.594044    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 09 21:13:10.599350 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:10.599298    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" event={"ID":"ebc421cd-26fa-41e4-81f1-f8783acb28e1","Type":"ContainerStarted","Data":"d42a07bc0be49e8d8c095e56aeebb98dd849d0fa176dd399d1ba0fa1d86bbfc7"}
Sep 09 21:13:10.917293 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:10.917208    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:13:10.917491 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:10.917473    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 09 21:13:10.917540 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:10.917515    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 09 21:13:11.917691 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:11.917659    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 09 21:13:11.918072 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:11.917721    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 09 21:13:12.917574 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:12.917477    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 09 21:13:12.917574 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:12.917548    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 09 21:13:13.917767 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:13.917730    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 09 21:13:13.918143 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:13.917784    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 09 21:13:14.917354 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:14.917294    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Sep 09 21:13:14.917527 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:14.917373    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Sep 09 21:13:15.920916 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:15.920886    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:13:15.921319 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:15.921249    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:13:15.921821 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:15.921802    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:13:17.531801 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:17.531748    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:22.817539 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:22.817513    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7_03820efb-7921-4926-8b09-2a62d1c2368e/payload-processing/0.log"
Sep 09 21:13:33.992249 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:33.992216    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"]
Sep 09 21:13:33.994786 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:33.994767    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:33.998049 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:33.998027    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7c0011\"/\"kube-root-ca.crt\""
Sep 09 21:13:33.998180 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:33.998027    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7c0011\"/\"ipp-route-7c0011-kserve-self-signed-certs\""
Sep 09 21:13:33.998180 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:33.998088    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-7c0011\"/\"default-dockercfg-pjjvn\""
Sep 09 21:13:33.998412 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:33.998394    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-7c0011\"/\"openshift-service-ca.crt\""
Sep 09 21:13:34.011468 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.011445    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"]
Sep 09 21:13:34.053297 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.053277    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:34.076470 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.076442    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqf58\" (UniqueName: \"kubernetes.io/projected/8338754a-83c4-4ba3-a368-10f2642ab473-kube-api-access-nqf58\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.076562 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.076477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8338754a-83c4-4ba3-a368-10f2642ab473-tls-certs\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.076562 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.076501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-home\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.076640 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.076600    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-dshm\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.076640 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.076622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-model-cache\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.076708 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.076643    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-tmp-dir\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.177810 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.177774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-dshm\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.177961 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.177817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-model-cache\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.177961 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.177913    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-tmp-dir\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.177961 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.177949    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqf58\" (UniqueName: \"kubernetes.io/projected/8338754a-83c4-4ba3-a368-10f2642ab473-kube-api-access-nqf58\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.178118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.177982    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8338754a-83c4-4ba3-a368-10f2642ab473-tls-certs\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.178118 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.178019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-home\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.178222 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.178153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-model-cache\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.178302 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.178278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-tmp-dir\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.178407 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.178299    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-home\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.180122 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.180095    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-dshm\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.180368 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.180351    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8338754a-83c4-4ba3-a368-10f2642ab473-tls-certs\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.189392 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.189371    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqf58\" (UniqueName: \"kubernetes.io/projected/8338754a-83c4-4ba3-a368-10f2642ab473-kube-api-access-nqf58\") pod \"ipp-route-7c0011-kserve-747d75699-zgfjz\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") " pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.305021 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.304964    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:34.581541 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.581329    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"]
Sep 09 21:13:34.583875 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:13:34.583849    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8338754a_83c4_4ba3_a368_10f2642ab473.slice/crio-3f5efdbc81ad5bcd44760a88b6b798321133262fab2b60a55b31c2e9032e5a20 WatchSource:0}: Error finding container 3f5efdbc81ad5bcd44760a88b6b798321133262fab2b60a55b31c2e9032e5a20: Status 404 returned error can't find the container with id 3f5efdbc81ad5bcd44760a88b6b798321133262fab2b60a55b31c2e9032e5a20
Sep 09 21:13:34.715267 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.715228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz" event={"ID":"8338754a-83c4-4ba3-a368-10f2642ab473","Type":"ContainerStarted","Data":"dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1"}
Sep 09 21:13:34.715446 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.715272    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz" event={"ID":"8338754a-83c4-4ba3-a368-10f2642ab473","Type":"ContainerStarted","Data":"3f5efdbc81ad5bcd44760a88b6b798321133262fab2b60a55b31c2e9032e5a20"}
Sep 09 21:13:34.738025 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:34.737978    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz" podStartSLOduration=1.73796292 podStartE2EDuration="1.73796292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:13:34.737440027 +0000 UTC m=+1994.929277628" watchObservedRunningTime="2026-09-09 21:13:34.73796292 +0000 UTC m=+1994.929800509"
Sep 09 21:13:37.675561 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:37.675522    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:13:44.305896 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:44.305864    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:44.306288 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:44.305908    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:44.318211 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:44.318186    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:44.776328 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:44.776291    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:13:46.824672 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:46.824635    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:52.508637 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:52.508597    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:13:58.948634 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:58.948605    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7_03820efb-7921-4926-8b09-2a62d1c2368e/payload-processing/0.log"
Sep 09 21:13:59.239631 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:59.239596    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:59.657358 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:59.657260    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:13:59.972826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:13:59.972738    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:03.133536 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:03.133498    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:03.243418 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:03.243386    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:04.312453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:04.312423    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:04.413463 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:04.413429    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:04.954486 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:04.954445    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:05.057874 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.057837    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:05.146049 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.146020    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"]
Sep 09 21:14:05.149339 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.149299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:05.151727 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.151713    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-5fc36f-dockercfg-vrvgq\""
Sep 09 21:14:05.152265 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.152243    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-5fc36f\""
Sep 09 21:14:05.165824 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.165804    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"]
Sep 09 21:14:05.232225 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.232194    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qv22m\" (UniqueName: \"kubernetes.io/projected/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-kube-api-access-qv22m\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54\" (UID: \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:05.232412 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.232233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54\" (UID: \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:05.332997 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.332955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qv22m\" (UniqueName: \"kubernetes.io/projected/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-kube-api-access-qv22m\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54\" (UID: \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:05.333505 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.333032    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54\" (UID: \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:05.333969 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.333943    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54\" (UID: \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:05.344963 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.344939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qv22m\" (UniqueName: \"kubernetes.io/projected/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-kube-api-access-qv22m\") pod \"payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54\" (UID: \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:05.459657 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.459627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:05.736708 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.736678    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"]
Sep 09 21:14:05.738992 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:14:05.738965    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaeb62b5_5d5c_47fe_9161_d6fe9f4076b5.slice/crio-a8eb35a0520fa7db1296cf2439fe57736bd1a387dfac4b3a2e7d4578d320771e WatchSource:0}: Error finding container a8eb35a0520fa7db1296cf2439fe57736bd1a387dfac4b3a2e7d4578d320771e: Status 404 returned error can't find the container with id a8eb35a0520fa7db1296cf2439fe57736bd1a387dfac4b3a2e7d4578d320771e
Sep 09 21:14:05.873642 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.873606    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54" event={"ID":"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5","Type":"ContainerStarted","Data":"052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f"}
Sep 09 21:14:05.873793 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.873657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54" event={"ID":"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5","Type":"ContainerStarted","Data":"a8eb35a0520fa7db1296cf2439fe57736bd1a387dfac4b3a2e7d4578d320771e"}
Sep 09 21:14:05.873793 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.873725    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:05.898417 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:05.898376    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54" podStartSLOduration=0.898360844 podStartE2EDuration="898.360844ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:14:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:14:05.897966642 +0000 UTC m=+2026.089804299" watchObservedRunningTime="2026-09-09 21:14:05.898360844 +0000 UTC m=+2026.090198443"
Sep 09 21:14:11.870572 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:11.870533    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:14:16.881485 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:16.881451    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:28.060531 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:28.060484    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:35.254589 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.254551    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:35.327195 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.327163    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"]
Sep 09 21:14:35.327448 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.327424    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54" podUID="daeb62b5-5d5c-47fe-9161-d6fe9f4076b5" containerName="payload-pre-processing" containerID="cri-o://052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f" gracePeriod=30
Sep 09 21:14:35.461519 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.461485    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:35.581972 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.581952    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:35.668754 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.668730    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qv22m\" (UniqueName: \"kubernetes.io/projected/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-kube-api-access-qv22m\") pod \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\" (UID: \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\") "
Sep 09 21:14:35.668894 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.668858    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-plugins-config-volume\") pod \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\" (UID: \"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5\") "
Sep 09 21:14:35.672381 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.669634    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "daeb62b5-5d5c-47fe-9161-d6fe9f4076b5" (UID: "daeb62b5-5d5c-47fe-9161-d6fe9f4076b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:14:35.672381 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.672010    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-kube-api-access-qv22m" (OuterVolumeSpecName: "kube-api-access-qv22m") pod "daeb62b5-5d5c-47fe-9161-d6fe9f4076b5" (UID: "daeb62b5-5d5c-47fe-9161-d6fe9f4076b5"). InnerVolumeSpecName "kube-api-access-qv22m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:14:35.740717 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.740681    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:35.770520 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.770497    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:14:35.770520 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.770515    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qv22m\" (UniqueName: \"kubernetes.io/projected/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5-kube-api-access-qv22m\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:14:35.878494 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:35.878390    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:36.026794 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.026764    2566 generic.go:356] "Generic (PLEG): container finished" podID="daeb62b5-5d5c-47fe-9161-d6fe9f4076b5" containerID="052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f" exitCode=0
Sep 09 21:14:36.026931 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.026837    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"
Sep 09 21:14:36.026931 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.026848    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54" event={"ID":"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5","Type":"ContainerDied","Data":"052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f"}
Sep 09 21:14:36.026931 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.026925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54" event={"ID":"daeb62b5-5d5c-47fe-9161-d6fe9f4076b5","Type":"ContainerDied","Data":"a8eb35a0520fa7db1296cf2439fe57736bd1a387dfac4b3a2e7d4578d320771e"}
Sep 09 21:14:36.027056 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.026948    2566 scope.go:117] "RemoveContainer" containerID="052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f"
Sep 09 21:14:36.036171 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.036151    2566 scope.go:117] "RemoveContainer" containerID="052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f"
Sep 09 21:14:36.036441 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:14:36.036418    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f\": container with ID starting with 052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f not found: ID does not exist" containerID="052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f"
Sep 09 21:14:36.036514 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.036451    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f"} err="failed to get container status \"052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f\": rpc error: code = NotFound desc = could not find container \"052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f\": container with ID starting with 052d6013506b1474a06201822d8aa11e608041f580fbc4a5d7ff65da857ed34f not found: ID does not exist"
Sep 09 21:14:36.061735 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.061713    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"]
Sep 09 21:14:36.070409 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.070379    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-5fc36f-d6f97ddcd-fcb54"]
Sep 09 21:14:36.152780 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.152712    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:36.286903 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:36.286872    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daeb62b5-5d5c-47fe-9161-d6fe9f4076b5" path="/var/lib/kubelet/pods/daeb62b5-5d5c-47fe-9161-d6fe9f4076b5/volumes"
Sep 09 21:14:39.791517 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:39.791484    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:14:42.350716 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:42.350679    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:42.451860 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:42.451828    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:42.620860 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:42.620751    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:46.266472 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:46.266439    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:14:59.263438 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:14:59.263404    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:15:00.202182 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.202152    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"]
Sep 09 21:15:00.202594 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.202580    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daeb62b5-5d5c-47fe-9161-d6fe9f4076b5" containerName="payload-pre-processing"
Sep 09 21:15:00.202646 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.202595    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="daeb62b5-5d5c-47fe-9161-d6fe9f4076b5" containerName="payload-pre-processing"
Sep 09 21:15:00.202682 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.202663    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="daeb62b5-5d5c-47fe-9161-d6fe9f4076b5" containerName="payload-pre-processing"
Sep 09 21:15:00.205632 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.205615    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"
Sep 09 21:15:00.211420 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.211392    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-glz44\""
Sep 09 21:15:00.215172 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.215156    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"
Sep 09 21:15:00.228921 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.226419    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t"]
Sep 09 21:15:00.230770 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.230720    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"]
Sep 09 21:15:00.230895 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.230806    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t"
Sep 09 21:15:00.252728 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.252677    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t"]
Sep 09 21:15:00.261737 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.260452    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t"
Sep 09 21:15:00.514564 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.514530    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"]
Sep 09 21:15:00.516040 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:15:00.516012    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c8f1d0_851f_41b4_89a8_41aed339fa2c.slice/crio-2ff8943d944a0c90bd7e975b77297d63169a085a9982727fec6136ab30bf89b0 WatchSource:0}: Error finding container 2ff8943d944a0c90bd7e975b77297d63169a085a9982727fec6136ab30bf89b0: Status 404 returned error can't find the container with id 2ff8943d944a0c90bd7e975b77297d63169a085a9982727fec6136ab30bf89b0
Sep 09 21:15:00.557217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:00.557192    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t"]
Sep 09 21:15:00.559604 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:15:00.559579    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a36ac3d_a918_4d17_80f7_21fe090b4dd8.slice/crio-89aa158844bbf821f1a3b9b2e8b8001d36d039b0bef46f4a4ea920d57758ad6a WatchSource:0}: Error finding container 89aa158844bbf821f1a3b9b2e8b8001d36d039b0bef46f4a4ea920d57758ad6a: Status 404 returned error can't find the container with id 89aa158844bbf821f1a3b9b2e8b8001d36d039b0bef46f4a4ea920d57758ad6a
Sep 09 21:15:01.158572 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.158484    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b" event={"ID":"59c8f1d0-851f-41b4-89a8-41aed339fa2c","Type":"ContainerStarted","Data":"69ccd7786377d370df31a5c4a38644518c6dd853eb311707749a1e9c6982b397"}
Sep 09 21:15:01.158572 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.158548    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b" event={"ID":"59c8f1d0-851f-41b4-89a8-41aed339fa2c","Type":"ContainerStarted","Data":"2ff8943d944a0c90bd7e975b77297d63169a085a9982727fec6136ab30bf89b0"}
Sep 09 21:15:01.159945 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.159922    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t" event={"ID":"7a36ac3d-a918-4d17-80f7-21fe090b4dd8","Type":"ContainerStarted","Data":"0293430fea96beb7359ef185cc410396e0c5c942f257b5ff8b9834b145d76e9c"}
Sep 09 21:15:01.160064 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.159958    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t" event={"ID":"7a36ac3d-a918-4d17-80f7-21fe090b4dd8","Type":"ContainerStarted","Data":"89aa158844bbf821f1a3b9b2e8b8001d36d039b0bef46f4a4ea920d57758ad6a"}
Sep 09 21:15:01.183376 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.183328    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b" podStartSLOduration=1.183295994 podStartE2EDuration="1.183295994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:01.181288614 +0000 UTC m=+2081.373126212" watchObservedRunningTime="2026-09-09 21:15:01.183295994 +0000 UTC m=+2081.375133594"
Sep 09 21:15:01.774233 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.774171    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t" podStartSLOduration=1.7741525280000001 podStartE2EDuration="1.774152528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:15:01.210129821 +0000 UTC m=+2081.401967419" watchObservedRunningTime="2026-09-09 21:15:01.774152528 +0000 UTC m=+2081.965990130"
Sep 09 21:15:01.774650 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.774480    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"]
Sep 09 21:15:01.824783 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.824751    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"]
Sep 09 21:15:01.824981 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.824956    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl" podUID="0af48d12-1074-456f-a35a-61037f0ca438" containerName="payload-pre-processing" containerID="cri-o://b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd" gracePeriod=30
Sep 09 21:15:01.860431 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.860381    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:01.954155 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:01.954123    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:02.063049 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.063007    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:02.077007 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.076987    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:15:02.162721 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.162679    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:02.168968 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.168938    2566 generic.go:356] "Generic (PLEG): container finished" podID="7a36ac3d-a918-4d17-80f7-21fe090b4dd8" containerID="0293430fea96beb7359ef185cc410396e0c5c942f257b5ff8b9834b145d76e9c" exitCode=0
Sep 09 21:15:02.169205 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.169031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t" event={"ID":"7a36ac3d-a918-4d17-80f7-21fe090b4dd8","Type":"ContainerDied","Data":"0293430fea96beb7359ef185cc410396e0c5c942f257b5ff8b9834b145d76e9c"}
Sep 09 21:15:02.170349 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.170329    2566 generic.go:356] "Generic (PLEG): container finished" podID="0af48d12-1074-456f-a35a-61037f0ca438" containerID="b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd" exitCode=0
Sep 09 21:15:02.170454 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.170363    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"
Sep 09 21:15:02.170454 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.170415    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl" event={"ID":"0af48d12-1074-456f-a35a-61037f0ca438","Type":"ContainerDied","Data":"b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd"}
Sep 09 21:15:02.170555 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.170464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl" event={"ID":"0af48d12-1074-456f-a35a-61037f0ca438","Type":"ContainerDied","Data":"edee51682709a6778ca529a491d929db2868067633020bc1d570c1f3d92d8bea"}
Sep 09 21:15:02.170555 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.170490    2566 scope.go:117] "RemoveContainer" containerID="b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd"
Sep 09 21:15:02.171973 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.171954    2566 generic.go:356] "Generic (PLEG): container finished" podID="59c8f1d0-851f-41b4-89a8-41aed339fa2c" containerID="69ccd7786377d370df31a5c4a38644518c6dd853eb311707749a1e9c6982b397" exitCode=0
Sep 09 21:15:02.172112 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.171990    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b" event={"ID":"59c8f1d0-851f-41b4-89a8-41aed339fa2c","Type":"ContainerDied","Data":"69ccd7786377d370df31a5c4a38644518c6dd853eb311707749a1e9c6982b397"}
Sep 09 21:15:02.178942 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.178922    2566 scope.go:117] "RemoveContainer" containerID="b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd"
Sep 09 21:15:02.179192 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:15:02.179174    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd\": container with ID starting with b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd not found: ID does not exist" containerID="b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd"
Sep 09 21:15:02.179246 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.179200    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd"} err="failed to get container status \"b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd\": rpc error: code = NotFound desc = could not find container \"b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd\": container with ID starting with b1d70e2a16c5fba92eafb877f7bbe6759db4093ef3b46006fab24c86091285cd not found: ID does not exist"
Sep 09 21:15:02.199486 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.199465    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pn6vf\" (UniqueName: \"kubernetes.io/projected/0af48d12-1074-456f-a35a-61037f0ca438-kube-api-access-pn6vf\") pod \"0af48d12-1074-456f-a35a-61037f0ca438\" (UID: \"0af48d12-1074-456f-a35a-61037f0ca438\") "
Sep 09 21:15:02.199597 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.199495    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af48d12-1074-456f-a35a-61037f0ca438-plugins-config-volume\") pod \"0af48d12-1074-456f-a35a-61037f0ca438\" (UID: \"0af48d12-1074-456f-a35a-61037f0ca438\") "
Sep 09 21:15:02.199926 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.199902    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af48d12-1074-456f-a35a-61037f0ca438-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0af48d12-1074-456f-a35a-61037f0ca438" (UID: "0af48d12-1074-456f-a35a-61037f0ca438"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:15:02.201478 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.201457    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af48d12-1074-456f-a35a-61037f0ca438-kube-api-access-pn6vf" (OuterVolumeSpecName: "kube-api-access-pn6vf") pod "0af48d12-1074-456f-a35a-61037f0ca438" (UID: "0af48d12-1074-456f-a35a-61037f0ca438"). InnerVolumeSpecName "kube-api-access-pn6vf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:02.300430 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.300403    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn6vf\" (UniqueName: \"kubernetes.io/projected/0af48d12-1074-456f-a35a-61037f0ca438-kube-api-access-pn6vf\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:15:02.300430 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.300428    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0af48d12-1074-456f-a35a-61037f0ca438-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:15:02.419512 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.419423    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:02.486697 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.486669    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"]
Sep 09 21:15:02.497739 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:02.497712    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7c0011-68cb8b69f4-cmnnl"]
Sep 09 21:15:03.314685 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:03.314658    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"
Sep 09 21:15:03.343885 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:03.343855    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t"
Sep 09 21:15:04.189750 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.189720    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t"
Sep 09 21:15:04.189925 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.189769    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t" event={"ID":"7a36ac3d-a918-4d17-80f7-21fe090b4dd8","Type":"ContainerDied","Data":"89aa158844bbf821f1a3b9b2e8b8001d36d039b0bef46f4a4ea920d57758ad6a"}
Sep 09 21:15:04.189925 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.189858    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89aa158844bbf821f1a3b9b2e8b8001d36d039b0bef46f4a4ea920d57758ad6a"
Sep 09 21:15:04.191089 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.191073    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"
Sep 09 21:15:04.191181 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.191069    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b" event={"ID":"59c8f1d0-851f-41b4-89a8-41aed339fa2c","Type":"ContainerDied","Data":"2ff8943d944a0c90bd7e975b77297d63169a085a9982727fec6136ab30bf89b0"}
Sep 09 21:15:04.191232 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.191215    2566 scope.go:117] "RemoveContainer" containerID="69ccd7786377d370df31a5c4a38644518c6dd853eb311707749a1e9c6982b397"
Sep 09 21:15:04.216875 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.216847    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"]
Sep 09 21:15:04.226109 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.226084    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-7c0011-29816475-cff7b"]
Sep 09 21:15:04.286869 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.286837    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0af48d12-1074-456f-a35a-61037f0ca438" path="/var/lib/kubelet/pods/0af48d12-1074-456f-a35a-61037f0ca438/volumes"
Sep 09 21:15:04.287138 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:04.287126    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59c8f1d0-851f-41b4-89a8-41aed339fa2c" path="/var/lib/kubelet/pods/59c8f1d0-851f-41b4-89a8-41aed339fa2c/volumes"
Sep 09 21:15:07.398848 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:07.398806    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"]
Sep 09 21:15:07.493342 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:07.493296    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:12.210063 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:12.210030    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:15:22.418409 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.418350    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz" podUID="8338754a-83c4-4ba3-a368-10f2642ab473" containerName="main" containerID="cri-o://dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1" gracePeriod=45
Sep 09 21:15:22.686525 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.686501    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:15:22.774043 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774018    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-model-cache\") pod \"8338754a-83c4-4ba3-a368-10f2642ab473\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") "
Sep 09 21:15:22.774214 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774059    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8338754a-83c4-4ba3-a368-10f2642ab473-tls-certs\") pod \"8338754a-83c4-4ba3-a368-10f2642ab473\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") "
Sep 09 21:15:22.774214 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774090    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-home\") pod \"8338754a-83c4-4ba3-a368-10f2642ab473\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") "
Sep 09 21:15:22.774214 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774117    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-tmp-dir\") pod \"8338754a-83c4-4ba3-a368-10f2642ab473\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") "
Sep 09 21:15:22.774477 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774231    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-dshm\") pod \"8338754a-83c4-4ba3-a368-10f2642ab473\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") "
Sep 09 21:15:22.774477 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774277    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqf58\" (UniqueName: \"kubernetes.io/projected/8338754a-83c4-4ba3-a368-10f2642ab473-kube-api-access-nqf58\") pod \"8338754a-83c4-4ba3-a368-10f2642ab473\" (UID: \"8338754a-83c4-4ba3-a368-10f2642ab473\") "
Sep 09 21:15:22.774477 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774323    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-model-cache" (OuterVolumeSpecName: "model-cache") pod "8338754a-83c4-4ba3-a368-10f2642ab473" (UID: "8338754a-83c4-4ba3-a368-10f2642ab473"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:22.774477 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774366    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8338754a-83c4-4ba3-a368-10f2642ab473" (UID: "8338754a-83c4-4ba3-a368-10f2642ab473"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:22.774477 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774431    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-home" (OuterVolumeSpecName: "home") pod "8338754a-83c4-4ba3-a368-10f2642ab473" (UID: "8338754a-83c4-4ba3-a368-10f2642ab473"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:22.774727 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774653    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:15:22.774727 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774669    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:15:22.774727 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.774678    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:15:22.776346 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.776298    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8338754a-83c4-4ba3-a368-10f2642ab473-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8338754a-83c4-4ba3-a368-10f2642ab473" (UID: "8338754a-83c4-4ba3-a368-10f2642ab473"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:15:22.776454 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.776381    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-dshm" (OuterVolumeSpecName: "dshm") pod "8338754a-83c4-4ba3-a368-10f2642ab473" (UID: "8338754a-83c4-4ba3-a368-10f2642ab473"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:15:22.776454 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.776431    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8338754a-83c4-4ba3-a368-10f2642ab473-kube-api-access-nqf58" (OuterVolumeSpecName: "kube-api-access-nqf58") pod "8338754a-83c4-4ba3-a368-10f2642ab473" (UID: "8338754a-83c4-4ba3-a368-10f2642ab473"). InnerVolumeSpecName "kube-api-access-nqf58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:15:22.875577 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.875552    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8338754a-83c4-4ba3-a368-10f2642ab473-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:15:22.875577 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.875576    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8338754a-83c4-4ba3-a368-10f2642ab473-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:15:22.875727 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:22.875585    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqf58\" (UniqueName: \"kubernetes.io/projected/8338754a-83c4-4ba3-a368-10f2642ab473-kube-api-access-nqf58\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:15:23.286917 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:23.286885    2566 generic.go:356] "Generic (PLEG): container finished" podID="8338754a-83c4-4ba3-a368-10f2642ab473" containerID="dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1" exitCode=0
Sep 09 21:15:23.287095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:23.286926    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"
Sep 09 21:15:23.287095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:23.286983    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz" event={"ID":"8338754a-83c4-4ba3-a368-10f2642ab473","Type":"ContainerDied","Data":"dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1"}
Sep 09 21:15:23.287095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:23.287024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz" event={"ID":"8338754a-83c4-4ba3-a368-10f2642ab473","Type":"ContainerDied","Data":"3f5efdbc81ad5bcd44760a88b6b798321133262fab2b60a55b31c2e9032e5a20"}
Sep 09 21:15:23.287095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:23.287042    2566 scope.go:117] "RemoveContainer" containerID="dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1"
Sep 09 21:15:23.296015 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:23.295999    2566 scope.go:117] "RemoveContainer" containerID="dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1"
Sep 09 21:15:23.296268 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:15:23.296250    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1\": container with ID starting with dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1 not found: ID does not exist" containerID="dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1"
Sep 09 21:15:23.296420 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:23.296277    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1"} err="failed to get container status \"dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1\": rpc error: code = NotFound desc = could not find container \"dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1\": container with ID starting with dfb6e925095e409491e6ff38dbe0d9a95d00934ff44ff995a28ead7e2ade99c1 not found: ID does not exist"
Sep 09 21:15:23.313363 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:23.313344    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"]
Sep 09 21:15:23.322710 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:23.322678    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-7c0011/ipp-route-7c0011-kserve-747d75699-zgfjz"]
Sep 09 21:15:24.287439 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:24.287407    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8338754a-83c4-4ba3-a368-10f2642ab473" path="/var/lib/kubelet/pods/8338754a-83c4-4ba3-a368-10f2642ab473/volumes"
Sep 09 21:15:43.926426 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:43.926386    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:45.018403 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:45.018361    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:51.045778 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:51.045740    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:56.979252 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:56.979221    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:57.095227 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:57.095191    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:57.372854 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:57.372765    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:57.478859 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:57.478824    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:15:58.587383 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:15:58.587340    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:08.330967 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.330927    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:08.368855 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.368822    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"]
Sep 09 21:16:08.369229 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.369175    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7" podUID="03820efb-7921-4926-8b09-2a62d1c2368e" containerName="payload-processing" containerID="cri-o://1fb38c31e8e49b37c94cf86bb6cffae8dd4726505dbd6c0b4eb2400c2ed7fb4f" gracePeriod=30
Sep 09 21:16:08.510261 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.510231    2566 generic.go:356] "Generic (PLEG): container finished" podID="03820efb-7921-4926-8b09-2a62d1c2368e" containerID="1fb38c31e8e49b37c94cf86bb6cffae8dd4726505dbd6c0b4eb2400c2ed7fb4f" exitCode=0
Sep 09 21:16:08.510434 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.510304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7" event={"ID":"03820efb-7921-4926-8b09-2a62d1c2368e","Type":"ContainerDied","Data":"1fb38c31e8e49b37c94cf86bb6cffae8dd4726505dbd6c0b4eb2400c2ed7fb4f"}
Sep 09 21:16:08.540283 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.540111    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:08.618506 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.618482    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:16:08.635145 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.635114    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:08.681615 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.681579    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6882\" (UniqueName: \"kubernetes.io/projected/03820efb-7921-4926-8b09-2a62d1c2368e-kube-api-access-k6882\") pod \"03820efb-7921-4926-8b09-2a62d1c2368e\" (UID: \"03820efb-7921-4926-8b09-2a62d1c2368e\") "
Sep 09 21:16:08.681765 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.681670    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03820efb-7921-4926-8b09-2a62d1c2368e-plugins-config-volume\") pod \"03820efb-7921-4926-8b09-2a62d1c2368e\" (UID: \"03820efb-7921-4926-8b09-2a62d1c2368e\") "
Sep 09 21:16:08.682012 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.681990    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03820efb-7921-4926-8b09-2a62d1c2368e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03820efb-7921-4926-8b09-2a62d1c2368e" (UID: "03820efb-7921-4926-8b09-2a62d1c2368e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:08.683640 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.683614    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03820efb-7921-4926-8b09-2a62d1c2368e-kube-api-access-k6882" (OuterVolumeSpecName: "kube-api-access-k6882") pod "03820efb-7921-4926-8b09-2a62d1c2368e" (UID: "03820efb-7921-4926-8b09-2a62d1c2368e"). InnerVolumeSpecName "kube-api-access-k6882". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:08.738401 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.738371    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:08.782555 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.782527    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03820efb-7921-4926-8b09-2a62d1c2368e-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:08.782555 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:08.782550    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6882\" (UniqueName: \"kubernetes.io/projected/03820efb-7921-4926-8b09-2a62d1c2368e-kube-api-access-k6882\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:09.057201 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:09.057167    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:09.516406 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:09.516378    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"
Sep 09 21:16:09.516832 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:09.516379    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7" event={"ID":"03820efb-7921-4926-8b09-2a62d1c2368e","Type":"ContainerDied","Data":"3a566af8725e70356485b8a32771d1619f9eac1a97fa70d90c872e817c257d8a"}
Sep 09 21:16:09.516832 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:09.516515    2566 scope.go:117] "RemoveContainer" containerID="1fb38c31e8e49b37c94cf86bb6cffae8dd4726505dbd6c0b4eb2400c2ed7fb4f"
Sep 09 21:16:09.562321 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:09.562284    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"]
Sep 09 21:16:09.565156 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:09.565126    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-7d51a3-54584bdd45-cxmz7"]
Sep 09 21:16:10.285697 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:10.285665    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03820efb-7921-4926-8b09-2a62d1c2368e" path="/var/lib/kubelet/pods/03820efb-7921-4926-8b09-2a62d1c2368e/volumes"
Sep 09 21:16:11.865083 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:11.865048    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:16:15.179405 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:15.179329    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:15.272060 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:15.272013    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:15.425345 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:15.425301    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"]
Sep 09 21:16:15.425614 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:15.425585    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy" containerID="cri-o://d42a07bc0be49e8d8c095e56aeebb98dd849d0fa176dd399d1ba0fa1d86bbfc7" gracePeriod=30
Sep 09 21:16:15.473681 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:15.473607    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:15.898013 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:15.897974    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:15.923541 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:15.923504    2566 patch_prober.go:28] interesting pod/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 09 21:16:15.923674 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:15.923573    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 09 21:16:20.576301 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.576260    2566 generic.go:356] "Generic (PLEG): container finished" podID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerID="d42a07bc0be49e8d8c095e56aeebb98dd849d0fa176dd399d1ba0fa1d86bbfc7" exitCode=0
Sep 09 21:16:20.576700 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.576349    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" event={"ID":"ebc421cd-26fa-41e4-81f1-f8783acb28e1","Type":"ContainerDied","Data":"d42a07bc0be49e8d8c095e56aeebb98dd849d0fa176dd399d1ba0fa1d86bbfc7"}
Sep 09 21:16:20.576700 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.576421    2566 scope.go:117] "RemoveContainer" containerID="ff9b50e57307a21568a036f8fc37bf05c66aee29be4aa0ffb875e6e7fbd78fbd"
Sep 09 21:16:20.673480 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.673453    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:16:20.786189 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786161    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-envoy\") pod \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") "
Sep 09 21:16:20.786388 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786199    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-socket\") pod \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") "
Sep 09 21:16:20.786388 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786222    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istiod-ca-cert\") pod \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") "
Sep 09 21:16:20.786388 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786239    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-certs\") pod \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") "
Sep 09 21:16:20.786388 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786289    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-credential-socket\") pod \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") "
Sep 09 21:16:20.786388 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786328    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-data\") pod \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") "
Sep 09 21:16:20.786677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786391    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-podinfo\") pod \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") "
Sep 09 21:16:20.786677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786433    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hxxd\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-kube-api-access-4hxxd\") pod \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") "
Sep 09 21:16:20.786677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786475    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-token\") pod \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\" (UID: \"ebc421cd-26fa-41e4-81f1-f8783acb28e1\") "
Sep 09 21:16:20.786677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786499    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ebc421cd-26fa-41e4-81f1-f8783acb28e1" (UID: "ebc421cd-26fa-41e4-81f1-f8783acb28e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:20.786677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786642    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ebc421cd-26fa-41e4-81f1-f8783acb28e1" (UID: "ebc421cd-26fa-41e4-81f1-f8783acb28e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:20.786677 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ebc421cd-26fa-41e4-81f1-f8783acb28e1" (UID: "ebc421cd-26fa-41e4-81f1-f8783acb28e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:20.786973 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786683    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ebc421cd-26fa-41e4-81f1-f8783acb28e1" (UID: "ebc421cd-26fa-41e4-81f1-f8783acb28e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:20.786973 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786786    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.786973 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786805    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istiod-ca-cert\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.786973 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786820    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-workload-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.786973 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786828    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-credential-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.786973 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.786907    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-data" (OuterVolumeSpecName: "istio-data") pod "ebc421cd-26fa-41e4-81f1-f8783acb28e1" (UID: "ebc421cd-26fa-41e4-81f1-f8783acb28e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:20.788491 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.788462    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ebc421cd-26fa-41e4-81f1-f8783acb28e1" (UID: "ebc421cd-26fa-41e4-81f1-f8783acb28e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:20.788605 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.788526    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ebc421cd-26fa-41e4-81f1-f8783acb28e1" (UID: "ebc421cd-26fa-41e4-81f1-f8783acb28e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:16:20.788605 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.788569    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-kube-api-access-4hxxd" (OuterVolumeSpecName: "kube-api-access-4hxxd") pod "ebc421cd-26fa-41e4-81f1-f8783acb28e1" (UID: "ebc421cd-26fa-41e4-81f1-f8783acb28e1"). InnerVolumeSpecName "kube-api-access-4hxxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:20.788706 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.788627    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-token" (OuterVolumeSpecName: "istio-token") pod "ebc421cd-26fa-41e4-81f1-f8783acb28e1" (UID: "ebc421cd-26fa-41e4-81f1-f8783acb28e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:20.887408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.887365    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-podinfo\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.887408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.887391    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hxxd\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-kube-api-access-4hxxd\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.887408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.887402    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-token\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.887408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.887410    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-envoy\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:20.887408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:20.887418    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ebc421cd-26fa-41e4-81f1-f8783acb28e1-istio-data\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:21.582481 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:21.582450    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"
Sep 09 21:16:21.582481 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:21.582460    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw" event={"ID":"ebc421cd-26fa-41e4-81f1-f8783acb28e1","Type":"ContainerDied","Data":"7f79a9bab2362b3c6e27bd9adf101d1240ea248ab232525d8c63ba6d02c80e31"}
Sep 09 21:16:21.582977 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:21.582514    2566 scope.go:117] "RemoveContainer" containerID="d42a07bc0be49e8d8c095e56aeebb98dd849d0fa176dd399d1ba0fa1d86bbfc7"
Sep 09 21:16:21.610984 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:21.610926    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"]
Sep 09 21:16:21.615275 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:21.615253    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-7d51a3-openshift-default-857c487d78-s7kxw"]
Sep 09 21:16:22.286361 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:22.286329    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" path="/var/lib/kubelet/pods/ebc421cd-26fa-41e4-81f1-f8783acb28e1/volumes"
Sep 09 21:16:26.402753 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:26.402710    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:27.715172 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:27.715137    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:29.376593 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:29.376565    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:16:33.574250 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:33.574220    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:33.669963 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:33.669930    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:33.817269 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:33.817236    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"]
Sep 09 21:16:33.817522 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:33.817495    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy" containerID="cri-o://db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e" gracePeriod=30
Sep 09 21:16:33.859658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:33.859582    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:34.302087 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:34.302050    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:39.085090 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.085041    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:16:39.230688 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230656    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-socket\") pod \"fff74b49-3c3f-473a-a02c-6e72b5079028\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") "
Sep 09 21:16:39.230851 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230692    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-964bv\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-kube-api-access-964bv\") pod \"fff74b49-3c3f-473a-a02c-6e72b5079028\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") "
Sep 09 21:16:39.230851 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230730    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff74b49-3c3f-473a-a02c-6e72b5079028-istiod-ca-cert\") pod \"fff74b49-3c3f-473a-a02c-6e72b5079028\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") "
Sep 09 21:16:39.230851 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230786    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-envoy\") pod \"fff74b49-3c3f-473a-a02c-6e72b5079028\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") "
Sep 09 21:16:39.230851 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230837    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-podinfo\") pod \"fff74b49-3c3f-473a-a02c-6e72b5079028\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") "
Sep 09 21:16:39.231088 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230870    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-credential-socket\") pod \"fff74b49-3c3f-473a-a02c-6e72b5079028\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") "
Sep 09 21:16:39.231088 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230895    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-data\") pod \"fff74b49-3c3f-473a-a02c-6e72b5079028\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") "
Sep 09 21:16:39.231088 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230924    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-token\") pod \"fff74b49-3c3f-473a-a02c-6e72b5079028\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") "
Sep 09 21:16:39.231088 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-certs\") pod \"fff74b49-3c3f-473a-a02c-6e72b5079028\" (UID: \"fff74b49-3c3f-473a-a02c-6e72b5079028\") "
Sep 09 21:16:39.231088 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.230960    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fff74b49-3c3f-473a-a02c-6e72b5079028" (UID: "fff74b49-3c3f-473a-a02c-6e72b5079028"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:39.231408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.231116    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fff74b49-3c3f-473a-a02c-6e72b5079028" (UID: "fff74b49-3c3f-473a-a02c-6e72b5079028"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:39.231408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.231203    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-data" (OuterVolumeSpecName: "istio-data") pod "fff74b49-3c3f-473a-a02c-6e72b5079028" (UID: "fff74b49-3c3f-473a-a02c-6e72b5079028"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:39.231408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.231223    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff74b49-3c3f-473a-a02c-6e72b5079028-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fff74b49-3c3f-473a-a02c-6e72b5079028" (UID: "fff74b49-3c3f-473a-a02c-6e72b5079028"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:39.231408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.231280    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-credential-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:39.231408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.231303    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-socket\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:39.231408 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.231387    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fff74b49-3c3f-473a-a02c-6e72b5079028" (UID: "fff74b49-3c3f-473a-a02c-6e72b5079028"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:39.232925 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.232893    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-kube-api-access-964bv" (OuterVolumeSpecName: "kube-api-access-964bv") pod "fff74b49-3c3f-473a-a02c-6e72b5079028" (UID: "fff74b49-3c3f-473a-a02c-6e72b5079028"). InnerVolumeSpecName "kube-api-access-964bv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:39.233034 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.232933    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-token" (OuterVolumeSpecName: "istio-token") pod "fff74b49-3c3f-473a-a02c-6e72b5079028" (UID: "fff74b49-3c3f-473a-a02c-6e72b5079028"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:39.233337 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.233288    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fff74b49-3c3f-473a-a02c-6e72b5079028" (UID: "fff74b49-3c3f-473a-a02c-6e72b5079028"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:16:39.233337 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.233295    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fff74b49-3c3f-473a-a02c-6e72b5079028" (UID: "fff74b49-3c3f-473a-a02c-6e72b5079028"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 09 21:16:39.332599 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.332578    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fff74b49-3c3f-473a-a02c-6e72b5079028-istiod-ca-cert\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:39.332599 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.332598    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-envoy\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:39.332723 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.332606    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-podinfo\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:39.332723 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.332615    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-data\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:39.332723 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.332623    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-istio-token\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:39.332723 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.332630    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fff74b49-3c3f-473a-a02c-6e72b5079028-workload-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:39.332723 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.332638    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-964bv\" (UniqueName: \"kubernetes.io/projected/fff74b49-3c3f-473a-a02c-6e72b5079028-kube-api-access-964bv\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:39.667694 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.667658    2566 generic.go:356] "Generic (PLEG): container finished" podID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerID="db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e" exitCode=0
Sep 09 21:16:39.667854 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.667708    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"
Sep 09 21:16:39.667854 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.667747    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" event={"ID":"fff74b49-3c3f-473a-a02c-6e72b5079028","Type":"ContainerDied","Data":"db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e"}
Sep 09 21:16:39.667854 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.667800    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6" event={"ID":"fff74b49-3c3f-473a-a02c-6e72b5079028","Type":"ContainerDied","Data":"21615d6a79ed97877fed6978d4293b236e605a2694e60b8c0134f31e77394390"}
Sep 09 21:16:39.667854 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.667824    2566 scope.go:117] "RemoveContainer" containerID="db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e"
Sep 09 21:16:39.676789 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.676772    2566 scope.go:117] "RemoveContainer" containerID="370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8"
Sep 09 21:16:39.684254 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.684238    2566 scope.go:117] "RemoveContainer" containerID="db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e"
Sep 09 21:16:39.684537 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:16:39.684513    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e\": container with ID starting with db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e not found: ID does not exist" containerID="db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e"
Sep 09 21:16:39.684618 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.684544    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e"} err="failed to get container status \"db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e\": rpc error: code = NotFound desc = could not find container \"db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e\": container with ID starting with db9065be561998de89e460c375e1f0aaad2f6d3e66be11fc476e9ba12406334e not found: ID does not exist"
Sep 09 21:16:39.684618 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.684561    2566 scope.go:117] "RemoveContainer" containerID="370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8"
Sep 09 21:16:39.684795 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:16:39.684779    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8\": container with ID starting with 370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8 not found: ID does not exist" containerID="370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8"
Sep 09 21:16:39.684850 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.684800    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8"} err="failed to get container status \"370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8\": rpc error: code = NotFound desc = could not find container \"370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8\": container with ID starting with 370b7553f0c8776ccbe6371d2b19e1919c82318349b64fb30bec82e2a5588bc8 not found: ID does not exist"
Sep 09 21:16:39.693933 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.693911    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"]
Sep 09 21:16:39.701391 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:39.701373    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-615c6f-openshift-default-9c8ddb9fd-6sfz6"]
Sep 09 21:16:40.286197 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:40.286167    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" path="/var/lib/kubelet/pods/fff74b49-3c3f-473a-a02c-6e72b5079028/volumes"
Sep 09 21:16:44.790384 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:44.790343    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t"]
Sep 09 21:16:44.791749 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:44.791727    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-0b5d4c-29816475-jxj5t"]
Sep 09 21:16:44.793949 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:44.793920    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:44.821975 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:44.821947    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"]
Sep 09 21:16:44.822205 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:44.822175    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz" podUID="cb933b86-35e9-4af2-8ddc-9f6211006896" containerName="payload-processing" containerID="cri-o://44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff" gracePeriod=30
Sep 09 21:16:44.826341 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:44.825956    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"]
Sep 09 21:16:44.826341 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:44.826173    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8" podUID="224bf361-04d9-4569-9857-aeced3bd1cd4" containerName="payload-pre-processing" containerID="cri-o://64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc" gracePeriod=30
Sep 09 21:16:44.878296 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:44.878260    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:44.978893 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:44.978847    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:45.078538 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.078503    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:45.098416 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.098391    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:16:45.100946 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.100926    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:16:45.178423 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.178395    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb933b86-35e9-4af2-8ddc-9f6211006896-plugins-config-volume\") pod \"cb933b86-35e9-4af2-8ddc-9f6211006896\" (UID: \"cb933b86-35e9-4af2-8ddc-9f6211006896\") "
Sep 09 21:16:45.178612 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.178447    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qccb7\" (UniqueName: \"kubernetes.io/projected/224bf361-04d9-4569-9857-aeced3bd1cd4-kube-api-access-qccb7\") pod \"224bf361-04d9-4569-9857-aeced3bd1cd4\" (UID: \"224bf361-04d9-4569-9857-aeced3bd1cd4\") "
Sep 09 21:16:45.178612 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.178487    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bf361-04d9-4569-9857-aeced3bd1cd4-plugins-config-volume\") pod \"224bf361-04d9-4569-9857-aeced3bd1cd4\" (UID: \"224bf361-04d9-4569-9857-aeced3bd1cd4\") "
Sep 09 21:16:45.178612 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.178519    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t2kb2\" (UniqueName: \"kubernetes.io/projected/cb933b86-35e9-4af2-8ddc-9f6211006896-kube-api-access-t2kb2\") pod \"cb933b86-35e9-4af2-8ddc-9f6211006896\" (UID: \"cb933b86-35e9-4af2-8ddc-9f6211006896\") "
Sep 09 21:16:45.178846 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.178824    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb933b86-35e9-4af2-8ddc-9f6211006896-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cb933b86-35e9-4af2-8ddc-9f6211006896" (UID: "cb933b86-35e9-4af2-8ddc-9f6211006896"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:45.178929 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.178900    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/224bf361-04d9-4569-9857-aeced3bd1cd4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "224bf361-04d9-4569-9857-aeced3bd1cd4" (UID: "224bf361-04d9-4569-9857-aeced3bd1cd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:16:45.180720 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.180699    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/224bf361-04d9-4569-9857-aeced3bd1cd4-kube-api-access-qccb7" (OuterVolumeSpecName: "kube-api-access-qccb7") pod "224bf361-04d9-4569-9857-aeced3bd1cd4" (UID: "224bf361-04d9-4569-9857-aeced3bd1cd4"). InnerVolumeSpecName "kube-api-access-qccb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:45.180843 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.180751    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb933b86-35e9-4af2-8ddc-9f6211006896-kube-api-access-t2kb2" (OuterVolumeSpecName: "kube-api-access-t2kb2") pod "cb933b86-35e9-4af2-8ddc-9f6211006896" (UID: "cb933b86-35e9-4af2-8ddc-9f6211006896"). InnerVolumeSpecName "kube-api-access-t2kb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:45.280090 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.280063    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qccb7\" (UniqueName: \"kubernetes.io/projected/224bf361-04d9-4569-9857-aeced3bd1cd4-kube-api-access-qccb7\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:45.280090 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.280087    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/224bf361-04d9-4569-9857-aeced3bd1cd4-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:45.280244 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.280097    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2kb2\" (UniqueName: \"kubernetes.io/projected/cb933b86-35e9-4af2-8ddc-9f6211006896-kube-api-access-t2kb2\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:45.280244 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.280106    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb933b86-35e9-4af2-8ddc-9f6211006896-plugins-config-volume\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:45.403770 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.403707    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:45.697699 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.697620    2566 generic.go:356] "Generic (PLEG): container finished" podID="224bf361-04d9-4569-9857-aeced3bd1cd4" containerID="64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc" exitCode=0
Sep 09 21:16:45.697699 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.697661    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"
Sep 09 21:16:45.697899 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.697706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8" event={"ID":"224bf361-04d9-4569-9857-aeced3bd1cd4","Type":"ContainerDied","Data":"64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc"}
Sep 09 21:16:45.697899 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.697756    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8" event={"ID":"224bf361-04d9-4569-9857-aeced3bd1cd4","Type":"ContainerDied","Data":"ed0bc8a74996765ea66caebc22f518d876051fb35770959f153c72b20742f9ae"}
Sep 09 21:16:45.697899 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.697786    2566 scope.go:117] "RemoveContainer" containerID="64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc"
Sep 09 21:16:45.699048 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.699021    2566 generic.go:356] "Generic (PLEG): container finished" podID="cb933b86-35e9-4af2-8ddc-9f6211006896" containerID="44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff" exitCode=0
Sep 09 21:16:45.699177 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.699061    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz" event={"ID":"cb933b86-35e9-4af2-8ddc-9f6211006896","Type":"ContainerDied","Data":"44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff"}
Sep 09 21:16:45.699177 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.699067    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"
Sep 09 21:16:45.699177 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.699100    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz" event={"ID":"cb933b86-35e9-4af2-8ddc-9f6211006896","Type":"ContainerDied","Data":"649a5d6983bccfa5e21520c526f7e8f2f18e6022357b72e90627198818881399"}
Sep 09 21:16:45.706929 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.706845    2566 scope.go:117] "RemoveContainer" containerID="64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc"
Sep 09 21:16:45.707151 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:16:45.707133    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc\": container with ID starting with 64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc not found: ID does not exist" containerID="64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc"
Sep 09 21:16:45.707239 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.707162    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc"} err="failed to get container status \"64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc\": rpc error: code = NotFound desc = could not find container \"64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc\": container with ID starting with 64c0687f3edf8aaca88368d98eba2d58786dbe46e72070a8770be3ce349c0ebc not found: ID does not exist"
Sep 09 21:16:45.707239 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.707185    2566 scope.go:117] "RemoveContainer" containerID="44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff"
Sep 09 21:16:45.714885 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.714869    2566 scope.go:117] "RemoveContainer" containerID="44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff"
Sep 09 21:16:45.715129 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:16:45.715114    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff\": container with ID starting with 44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff not found: ID does not exist" containerID="44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff"
Sep 09 21:16:45.715177 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.715134    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff"} err="failed to get container status \"44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff\": rpc error: code = NotFound desc = could not find container \"44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff\": container with ID starting with 44ad38fb3039d096ab0d38aae60dba1cc79ee450e9d62ba01113ebb0676876ff not found: ID does not exist"
Sep 09 21:16:45.722921 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.722890    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"]
Sep 09 21:16:45.724545 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.724526    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-0b5d4c-58bf995786-llmv8"]
Sep 09 21:16:45.734690 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.734671    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"]
Sep 09 21:16:45.741636 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:45.741619    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-0b5d4c-64bcb644c7-4bqtz"]
Sep 09 21:16:46.288223 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:46.288194    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="224bf361-04d9-4569-9857-aeced3bd1cd4" path="/var/lib/kubelet/pods/224bf361-04d9-4569-9857-aeced3bd1cd4/volumes"
Sep 09 21:16:46.288773 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:46.288484    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a36ac3d-a918-4d17-80f7-21fe090b4dd8" path="/var/lib/kubelet/pods/7a36ac3d-a918-4d17-80f7-21fe090b4dd8/volumes"
Sep 09 21:16:46.288773 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:46.288749    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb933b86-35e9-4af2-8ddc-9f6211006896" path="/var/lib/kubelet/pods/cb933b86-35e9-4af2-8ddc-9f6211006896/volumes"
Sep 09 21:16:47.190109 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:47.190071    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:16:52.134505 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:52.134467    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:52.236755 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:52.236717    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:52.421986 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:52.421903    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:16:55.087533 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.087498    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-qzg8v"]
Sep 09 21:16:55.087949 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.087727    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v" podUID="7df9cab9-ee01-4b01-83a1-7d700b86c6aa" containerName="manager" containerID="cri-o://25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92" gracePeriod=30
Sep 09 21:16:55.324846 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.324823    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 21:16:55.459749 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.459680    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-webhook-cert\") pod \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") "
Sep 09 21:16:55.459891 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.459767    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-metrics-certs\") pod \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") "
Sep 09 21:16:55.459891 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.459832    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bzhqn\" (UniqueName: \"kubernetes.io/projected/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-kube-api-access-bzhqn\") pod \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\" (UID: \"7df9cab9-ee01-4b01-83a1-7d700b86c6aa\") "
Sep 09 21:16:55.461743 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.461717    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7df9cab9-ee01-4b01-83a1-7d700b86c6aa" (UID: "7df9cab9-ee01-4b01-83a1-7d700b86c6aa"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:55.461909 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.461894    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7df9cab9-ee01-4b01-83a1-7d700b86c6aa" (UID: "7df9cab9-ee01-4b01-83a1-7d700b86c6aa"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:16:55.462021 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.461998    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-kube-api-access-bzhqn" (OuterVolumeSpecName: "kube-api-access-bzhqn") pod "7df9cab9-ee01-4b01-83a1-7d700b86c6aa" (UID: "7df9cab9-ee01-4b01-83a1-7d700b86c6aa"). InnerVolumeSpecName "kube-api-access-bzhqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:16:55.560762 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.560734    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-metrics-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:55.560762 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.560759    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzhqn\" (UniqueName: \"kubernetes.io/projected/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-kube-api-access-bzhqn\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:55.560898 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.560774    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7df9cab9-ee01-4b01-83a1-7d700b86c6aa-webhook-cert\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:16:55.747217 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.747188    2566 generic.go:356] "Generic (PLEG): container finished" podID="7df9cab9-ee01-4b01-83a1-7d700b86c6aa" containerID="25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92" exitCode=0
Sep 09 21:16:55.747342 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.747233    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v"
Sep 09 21:16:55.747342 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.747236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v" event={"ID":"7df9cab9-ee01-4b01-83a1-7d700b86c6aa","Type":"ContainerDied","Data":"25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92"}
Sep 09 21:16:55.747429 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.747363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-qzg8v" event={"ID":"7df9cab9-ee01-4b01-83a1-7d700b86c6aa","Type":"ContainerDied","Data":"1dd7a96d1775258c194e6c601dfa84e333b4690fd021d027b37cb0c87bf872f8"}
Sep 09 21:16:55.747429 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.747398    2566 scope.go:117] "RemoveContainer" containerID="25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92"
Sep 09 21:16:55.757216 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.757199    2566 scope.go:117] "RemoveContainer" containerID="25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92"
Sep 09 21:16:55.757472 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:16:55.757453    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92\": container with ID starting with 25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92 not found: ID does not exist" containerID="25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92"
Sep 09 21:16:55.757525 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.757481    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92"} err="failed to get container status \"25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92\": rpc error: code = NotFound desc = could not find container \"25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92\": container with ID starting with 25a93ed4d79aa4fe7e1c424055759095568f623dbcf9bb631623096ccae8ef92 not found: ID does not exist"
Sep 09 21:16:55.771564 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.771541    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-qzg8v"]
Sep 09 21:16:55.777187 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:55.777167    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-qzg8v"]
Sep 09 21:16:56.285636 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:56.285605    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7df9cab9-ee01-4b01-83a1-7d700b86c6aa" path="/var/lib/kubelet/pods/7df9cab9-ee01-4b01-83a1-7d700b86c6aa/volumes"
Sep 09 21:16:57.396761 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:16:57.396727    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:17:56.587213 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587121    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-m29wv"]
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587618    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587642    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587657    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587666    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587675    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03820efb-7921-4926-8b09-2a62d1c2368e" containerName="payload-processing"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587684    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="03820efb-7921-4926-8b09-2a62d1c2368e" containerName="payload-processing"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587709    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="224bf361-04d9-4569-9857-aeced3bd1cd4" containerName="payload-pre-processing"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587717    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="224bf361-04d9-4569-9857-aeced3bd1cd4" containerName="payload-pre-processing"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587729    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a36ac3d-a918-4d17-80f7-21fe090b4dd8" containerName="cleanup"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587738    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a36ac3d-a918-4d17-80f7-21fe090b4dd8" containerName="cleanup"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587752    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7df9cab9-ee01-4b01-83a1-7d700b86c6aa" containerName="manager"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587760    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="7df9cab9-ee01-4b01-83a1-7d700b86c6aa" containerName="manager"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587771    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb933b86-35e9-4af2-8ddc-9f6211006896" containerName="payload-processing"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587780    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb933b86-35e9-4af2-8ddc-9f6211006896" containerName="payload-processing"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587789    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587799    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587807    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0af48d12-1074-456f-a35a-61037f0ca438" containerName="payload-pre-processing"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587815    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0af48d12-1074-456f-a35a-61037f0ca438" containerName="payload-pre-processing"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587822    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59c8f1d0-851f-41b4-89a8-41aed339fa2c" containerName="cleanup"
Sep 09 21:17:56.587826 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587830    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="59c8f1d0-851f-41b4-89a8-41aed339fa2c" containerName="cleanup"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587841    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587849    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587859    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8338754a-83c4-4ba3-a368-10f2642ab473" containerName="main"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587867    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8338754a-83c4-4ba3-a368-10f2642ab473" containerName="main"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587956    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587968    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0af48d12-1074-456f-a35a-61037f0ca438" containerName="payload-pre-processing"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587978    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587987    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7df9cab9-ee01-4b01-83a1-7d700b86c6aa" containerName="manager"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.587996    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebc421cd-26fa-41e4-81f1-f8783acb28e1" containerName="istio-proxy"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.588005    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb933b86-35e9-4af2-8ddc-9f6211006896" containerName="payload-processing"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.588013    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fff74b49-3c3f-473a-a02c-6e72b5079028" containerName="istio-proxy"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.588021    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8338754a-83c4-4ba3-a368-10f2642ab473" containerName="main"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.588031    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a36ac3d-a918-4d17-80f7-21fe090b4dd8" containerName="cleanup"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.588039    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="224bf361-04d9-4569-9857-aeced3bd1cd4" containerName="payload-pre-processing"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.588049    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="03820efb-7921-4926-8b09-2a62d1c2368e" containerName="payload-processing"
Sep 09 21:17:56.588815 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.588056    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="59c8f1d0-851f-41b4-89a8-41aed339fa2c" containerName="cleanup"
Sep 09 21:17:56.591075 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.591056    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.594509 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.594481    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 21:17:56.595464 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.595440    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mbhzx\""
Sep 09 21:17:56.595464 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.595453    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 21:17:56.595764 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.595738    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 21:17:56.606590 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.606566    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-m29wv"]
Sep 09 21:17:56.772548 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.772510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-webhook-cert\") pod \"maas-controller-86f6d44ffc-m29wv\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") " pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.772548 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.772550    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9g2z\" (UniqueName: \"kubernetes.io/projected/ea78d124-5324-4926-ab9c-b95e9554f72a-kube-api-access-w9g2z\") pod \"maas-controller-86f6d44ffc-m29wv\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") " pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.772771 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.772582    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-metrics-certs\") pod \"maas-controller-86f6d44ffc-m29wv\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") " pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.873018 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.872930    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-metrics-certs\") pod \"maas-controller-86f6d44ffc-m29wv\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") " pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.873018 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.873019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-webhook-cert\") pod \"maas-controller-86f6d44ffc-m29wv\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") " pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.873242 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.873044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9g2z\" (UniqueName: \"kubernetes.io/projected/ea78d124-5324-4926-ab9c-b95e9554f72a-kube-api-access-w9g2z\") pod \"maas-controller-86f6d44ffc-m29wv\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") " pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.875513 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.875478    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-webhook-cert\") pod \"maas-controller-86f6d44ffc-m29wv\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") " pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.875646 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.875579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-metrics-certs\") pod \"maas-controller-86f6d44ffc-m29wv\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") " pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.884894 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.884865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9g2z\" (UniqueName: \"kubernetes.io/projected/ea78d124-5324-4926-ab9c-b95e9554f72a-kube-api-access-w9g2z\") pod \"maas-controller-86f6d44ffc-m29wv\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") " pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:56.900939 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:56.900915    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:57.176248 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:57.176063    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-m29wv"]
Sep 09 21:17:57.178964 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:17:57.178936    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea78d124_5324_4926_ab9c_b95e9554f72a.slice/crio-565821c147b541e12d759f9c41f9ecccdde1844103ccd8abf030c91589471f33 WatchSource:0}: Error finding container 565821c147b541e12d759f9c41f9ecccdde1844103ccd8abf030c91589471f33: Status 404 returned error can't find the container with id 565821c147b541e12d759f9c41f9ecccdde1844103ccd8abf030c91589471f33
Sep 09 21:17:57.180443 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:57.180422    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:17:57.782240 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:57.782210    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:17:58.033446 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:58.033337    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-m29wv" event={"ID":"ea78d124-5324-4926-ab9c-b95e9554f72a","Type":"ContainerStarted","Data":"c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534"}
Sep 09 21:17:58.033604 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:58.033496    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:17:58.033604 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:58.033542    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-m29wv" event={"ID":"ea78d124-5324-4926-ab9c-b95e9554f72a","Type":"ContainerStarted","Data":"565821c147b541e12d759f9c41f9ecccdde1844103ccd8abf030c91589471f33"}
Sep 09 21:17:58.059684 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:17:58.059636    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86f6d44ffc-m29wv" podStartSLOduration=1.465228347 podStartE2EDuration="2.059621935s" totalImagesPullingTime="594.393588ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:17:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:17:58.059028164 +0000 UTC m=+2258.250865760" watchObservedRunningTime="2026-09-09 21:17:58.059621935 +0000 UTC m=+2258.251459531"
Sep 09 21:18:09.041229 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:09.041197    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:18:18.792841 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:18.792790    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:19.788758 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:19.788721    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:18:20.409919 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.409873    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:20.480836 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.480800    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-m29wv"]
Sep 09 21:18:20.481098 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.481072    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-86f6d44ffc-m29wv" podUID="ea78d124-5324-4926-ab9c-b95e9554f72a" containerName="manager" containerID="cri-o://c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534" gracePeriod=30
Sep 09 21:18:20.734235 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.734194    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:18:20.776800 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.776763    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-webhook-cert\") pod \"ea78d124-5324-4926-ab9c-b95e9554f72a\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") "
Sep 09 21:18:20.776982 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.776823    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9g2z\" (UniqueName: \"kubernetes.io/projected/ea78d124-5324-4926-ab9c-b95e9554f72a-kube-api-access-w9g2z\") pod \"ea78d124-5324-4926-ab9c-b95e9554f72a\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") "
Sep 09 21:18:20.776982 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.776844    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-metrics-certs\") pod \"ea78d124-5324-4926-ab9c-b95e9554f72a\" (UID: \"ea78d124-5324-4926-ab9c-b95e9554f72a\") "
Sep 09 21:18:20.778960 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.778925    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "ea78d124-5324-4926-ab9c-b95e9554f72a" (UID: "ea78d124-5324-4926-ab9c-b95e9554f72a"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:18:20.779095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.779019    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ea78d124-5324-4926-ab9c-b95e9554f72a" (UID: "ea78d124-5324-4926-ab9c-b95e9554f72a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:18:20.779095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.779016    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea78d124-5324-4926-ab9c-b95e9554f72a-kube-api-access-w9g2z" (OuterVolumeSpecName: "kube-api-access-w9g2z") pod "ea78d124-5324-4926-ab9c-b95e9554f72a" (UID: "ea78d124-5324-4926-ab9c-b95e9554f72a"). InnerVolumeSpecName "kube-api-access-w9g2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:18:20.877516 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.877484    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9g2z\" (UniqueName: \"kubernetes.io/projected/ea78d124-5324-4926-ab9c-b95e9554f72a-kube-api-access-w9g2z\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:18:20.877516 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.877511    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-metrics-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:18:20.877516 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:20.877519    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea78d124-5324-4926-ab9c-b95e9554f72a-webhook-cert\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:18:21.139189 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.139151    2566 generic.go:356] "Generic (PLEG): container finished" podID="ea78d124-5324-4926-ab9c-b95e9554f72a" containerID="c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534" exitCode=0
Sep 09 21:18:21.139417 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.139194    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-m29wv"
Sep 09 21:18:21.139417 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.139221    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-m29wv" event={"ID":"ea78d124-5324-4926-ab9c-b95e9554f72a","Type":"ContainerDied","Data":"c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534"}
Sep 09 21:18:21.139417 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.139269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-m29wv" event={"ID":"ea78d124-5324-4926-ab9c-b95e9554f72a","Type":"ContainerDied","Data":"565821c147b541e12d759f9c41f9ecccdde1844103ccd8abf030c91589471f33"}
Sep 09 21:18:21.139417 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.139294    2566 scope.go:117] "RemoveContainer" containerID="c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534"
Sep 09 21:18:21.148406 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.148388    2566 scope.go:117] "RemoveContainer" containerID="c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534"
Sep 09 21:18:21.148690 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:18:21.148670    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534\": container with ID starting with c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534 not found: ID does not exist" containerID="c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534"
Sep 09 21:18:21.148739 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.148699    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534"} err="failed to get container status \"c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534\": rpc error: code = NotFound desc = could not find container \"c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534\": container with ID starting with c75a5983c977362acb73d6b1b527010fb7d32c716666d1214ccec47bf9a93534 not found: ID does not exist"
Sep 09 21:18:21.164532 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.164502    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-m29wv"]
Sep 09 21:18:21.171894 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.171873    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-m29wv"]
Sep 09 21:18:21.843070 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.843033    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:21.955810 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.955772    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-c2v9z"]
Sep 09 21:18:21.956147 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.956134    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea78d124-5324-4926-ab9c-b95e9554f72a" containerName="manager"
Sep 09 21:18:21.956188 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.956149    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea78d124-5324-4926-ab9c-b95e9554f72a" containerName="manager"
Sep 09 21:18:21.956229 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.956220    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea78d124-5324-4926-ab9c-b95e9554f72a" containerName="manager"
Sep 09 21:18:21.960281 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.960265    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:21.969966 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.969945    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 09 21:18:21.970101 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.969968    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mbhzx\""
Sep 09 21:18:21.970101 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.969947    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 09 21:18:21.970101 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.970081    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 09 21:18:21.976900 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:21.976873    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-c2v9z"]
Sep 09 21:18:22.087483 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.087443    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3329450d-370d-40e1-aef5-f9e2fbbd2224-webhook-cert\") pod \"maas-controller-86f6d44ffc-c2v9z\" (UID: \"3329450d-370d-40e1-aef5-f9e2fbbd2224\") " pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.087681 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.087502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6zzz\" (UniqueName: \"kubernetes.io/projected/3329450d-370d-40e1-aef5-f9e2fbbd2224-kube-api-access-p6zzz\") pod \"maas-controller-86f6d44ffc-c2v9z\" (UID: \"3329450d-370d-40e1-aef5-f9e2fbbd2224\") " pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.087681 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.087560    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3329450d-370d-40e1-aef5-f9e2fbbd2224-metrics-certs\") pod \"maas-controller-86f6d44ffc-c2v9z\" (UID: \"3329450d-370d-40e1-aef5-f9e2fbbd2224\") " pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.188426 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.188302    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3329450d-370d-40e1-aef5-f9e2fbbd2224-webhook-cert\") pod \"maas-controller-86f6d44ffc-c2v9z\" (UID: \"3329450d-370d-40e1-aef5-f9e2fbbd2224\") " pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.188426 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.188382    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6zzz\" (UniqueName: \"kubernetes.io/projected/3329450d-370d-40e1-aef5-f9e2fbbd2224-kube-api-access-p6zzz\") pod \"maas-controller-86f6d44ffc-c2v9z\" (UID: \"3329450d-370d-40e1-aef5-f9e2fbbd2224\") " pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.188426 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.188412    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3329450d-370d-40e1-aef5-f9e2fbbd2224-metrics-certs\") pod \"maas-controller-86f6d44ffc-c2v9z\" (UID: \"3329450d-370d-40e1-aef5-f9e2fbbd2224\") " pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.190915 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.190890    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3329450d-370d-40e1-aef5-f9e2fbbd2224-webhook-cert\") pod \"maas-controller-86f6d44ffc-c2v9z\" (UID: \"3329450d-370d-40e1-aef5-f9e2fbbd2224\") " pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.191053 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.190899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3329450d-370d-40e1-aef5-f9e2fbbd2224-metrics-certs\") pod \"maas-controller-86f6d44ffc-c2v9z\" (UID: \"3329450d-370d-40e1-aef5-f9e2fbbd2224\") " pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.198353 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.198295    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6zzz\" (UniqueName: \"kubernetes.io/projected/3329450d-370d-40e1-aef5-f9e2fbbd2224-kube-api-access-p6zzz\") pod \"maas-controller-86f6d44ffc-c2v9z\" (UID: \"3329450d-370d-40e1-aef5-f9e2fbbd2224\") " pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.270099 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.270065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:22.287380 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.287354    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea78d124-5324-4926-ab9c-b95e9554f72a" path="/var/lib/kubelet/pods/ea78d124-5324-4926-ab9c-b95e9554f72a/volumes"
Sep 09 21:18:22.542777 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:22.542748    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-86f6d44ffc-c2v9z"]
Sep 09 21:18:22.546410 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:18:22.546383    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3329450d_370d_40e1_aef5_f9e2fbbd2224.slice/crio-2bfb85b1c8435be8d4ce314938e9d19bcb2c18394ba22fa05f200278065cf453 WatchSource:0}: Error finding container 2bfb85b1c8435be8d4ce314938e9d19bcb2c18394ba22fa05f200278065cf453: Status 404 returned error can't find the container with id 2bfb85b1c8435be8d4ce314938e9d19bcb2c18394ba22fa05f200278065cf453
Sep 09 21:18:23.153618 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:23.153576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-c2v9z" event={"ID":"3329450d-370d-40e1-aef5-f9e2fbbd2224","Type":"ContainerStarted","Data":"4156d6faf7c58f3eab46fca2261076bf19a0b85819ab791783fb91ef47258a99"}
Sep 09 21:18:23.154020 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:23.153629    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-86f6d44ffc-c2v9z" event={"ID":"3329450d-370d-40e1-aef5-f9e2fbbd2224","Type":"ContainerStarted","Data":"2bfb85b1c8435be8d4ce314938e9d19bcb2c18394ba22fa05f200278065cf453"}
Sep 09 21:18:23.154020 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:23.153705    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:23.176685 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:23.176587    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-86f6d44ffc-c2v9z" podStartSLOduration=1.807825038 podStartE2EDuration="2.176572274s" totalImagesPullingTime="368.747236ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:18:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:18:23.175049166 +0000 UTC m=+2283.366886765" watchObservedRunningTime="2026-09-09 21:18:23.176572274 +0000 UTC m=+2283.368409910"
Sep 09 21:18:24.587101 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:24.587064    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:18:34.160768 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:34.160735    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-86f6d44ffc-c2v9z"
Sep 09 21:18:46.487605 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:46.487566    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:18:54.764453 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:54.764416    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:18:56.203525 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:18:56.203494    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:19:00.872967 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:19:00.872930    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:19:08.963921 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:19:08.963884    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:19:19.874449 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:19:19.874369    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:19:29.758250 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:19:29.758216    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:20:14.383372 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:20:14.383336    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:47.574077 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:20:47.573986    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:47.684466 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:20:47.684433    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:48.161463 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:20:48.161424    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:59.048716 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:20:59.048678    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:20:59.822538 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:20:59.822503    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:21:07.889778 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:07.889745    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:21:13.585071 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.585041    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"]
Sep 09 21:21:13.587868 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.587843    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.591210 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.591187    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-4b9eccda-kserve-self-signed-certs\""
Sep 09 21:21:13.602066 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.602042    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"]
Sep 09 21:21:13.633750 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.633716    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:21:13.688988 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.688965    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-home\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.689123 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.689006    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tls-certs\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.689123 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.689026    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-dshm\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.689244 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.689124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-model-cache\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.689244 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.689174    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tmp-dir\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.689244 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.689216    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlzpv\" (UniqueName: \"kubernetes.io/projected/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-kube-api-access-qlzpv\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.790108 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.790076    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-model-cache\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.790108 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.790110    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tmp-dir\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.790345 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.790132    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlzpv\" (UniqueName: \"kubernetes.io/projected/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-kube-api-access-qlzpv\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.790345 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.790181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-home\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.790345 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.790269    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tls-certs\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.790345 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.790297    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-dshm\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.790564 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.790492    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-model-cache\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.790564 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.790542    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tmp-dir\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.790633 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.790572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-home\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.792599 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.792573    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-dshm\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.792738 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.792720    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tls-certs\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.800876 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.800852    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlzpv\" (UniqueName: \"kubernetes.io/projected/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-kube-api-access-qlzpv\") pod \"e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") " pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:13.897592 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:13.897535    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:14.175992 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:14.175967    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"]
Sep 09 21:21:14.178399 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:21:14.178374    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod31ba19cd_25b7_4e3b_9f40_67cbd687ae88.slice/crio-3d660fd4575075f72bbaf5e54980c01edae9afb28a8450e5cd7288cfe7f2dc74 WatchSource:0}: Error finding container 3d660fd4575075f72bbaf5e54980c01edae9afb28a8450e5cd7288cfe7f2dc74: Status 404 returned error can't find the container with id 3d660fd4575075f72bbaf5e54980c01edae9afb28a8450e5cd7288cfe7f2dc74
Sep 09 21:21:14.932428 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:14.932382    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf" event={"ID":"31ba19cd-25b7-4e3b-9f40-67cbd687ae88","Type":"ContainerStarted","Data":"d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861"}
Sep 09 21:21:14.932428 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:14.932429    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf" event={"ID":"31ba19cd-25b7-4e3b-9f40-67cbd687ae88","Type":"ContainerStarted","Data":"3d660fd4575075f72bbaf5e54980c01edae9afb28a8450e5cd7288cfe7f2dc74"}
Sep 09 21:21:14.957164 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:14.957115    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf" podStartSLOduration=1.9571010420000001 podStartE2EDuration="1.957101042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:21:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:21:14.955188489 +0000 UTC m=+2455.147026088" watchObservedRunningTime="2026-09-09 21:21:14.957101042 +0000 UTC m=+2455.148938641"
Sep 09 21:21:15.869900 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:15.869866    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:21:20.709138 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:20.709100    2566 scope.go:117] "RemoveContainer" containerID="0293430fea96beb7359ef185cc410396e0c5c942f257b5ff8b9834b145d76e9c"
Sep 09 21:21:23.898278 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:23.898237    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:23.898278 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:23.898280    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:23.910899 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:23.910869    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:23.983747 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:23.983722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:21:34.094416 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.094376    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"]
Sep 09 21:21:34.097614 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.097592    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.100504 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.100486    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-4b9eccda-kserve-self-signed-certs\""
Sep 09 21:21:34.111792 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.111772    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"]
Sep 09 21:21:34.145190 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.145158    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:21:34.259738 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.259701    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/1f21536b-a736-4fd6-804f-dd59462c79e2-kube-api-access-qs6qt\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.259921 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.259753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f21536b-a736-4fd6-804f-dd59462c79e2-tls-certs\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.259921 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.259849    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-tmp-dir\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.259921 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.259882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-dshm\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.259921 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.259905    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-model-cache\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.260075 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.259958    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-home\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.360500 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.360407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-tmp-dir\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.360500 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.360447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-dshm\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.360500 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.360466    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-model-cache\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.360500 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.360484    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-home\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.360832 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.360516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/1f21536b-a736-4fd6-804f-dd59462c79e2-kube-api-access-qs6qt\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.360832 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.360542    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f21536b-a736-4fd6-804f-dd59462c79e2-tls-certs\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.360931 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.360863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-model-cache\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.360931 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.360892    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-home\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.361025 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.361001    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-tmp-dir\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.362790 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.362767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-dshm\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.363045 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.363027    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f21536b-a736-4fd6-804f-dd59462c79e2-tls-certs\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.369537 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.369512    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/1f21536b-a736-4fd6-804f-dd59462c79e2-kube-api-access-qs6qt\") pod \"e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") " pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.406552 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.406524    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:34.692803 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:34.692724    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"]
Sep 09 21:21:34.695965 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:21:34.695931    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1f21536b_a736_4fd6_804f_dd59462c79e2.slice/crio-4402031fbc772bbbd6a934e166f1f2f7fe085f4951ed9f4ca0a0e1cae7ada6ef WatchSource:0}: Error finding container 4402031fbc772bbbd6a934e166f1f2f7fe085f4951ed9f4ca0a0e1cae7ada6ef: Status 404 returned error can't find the container with id 4402031fbc772bbbd6a934e166f1f2f7fe085f4951ed9f4ca0a0e1cae7ada6ef
Sep 09 21:21:35.026880 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:35.026850    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf" event={"ID":"1f21536b-a736-4fd6-804f-dd59462c79e2","Type":"ContainerStarted","Data":"04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6"}
Sep 09 21:21:35.027069 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:35.026889    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf" event={"ID":"1f21536b-a736-4fd6-804f-dd59462c79e2","Type":"ContainerStarted","Data":"4402031fbc772bbbd6a934e166f1f2f7fe085f4951ed9f4ca0a0e1cae7ada6ef"}
Sep 09 21:21:35.048778 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:35.048737    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf" podStartSLOduration=1.048722836 podStartE2EDuration="1.048722836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:21:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:21:35.046532427 +0000 UTC m=+2475.238370037" watchObservedRunningTime="2026-09-09 21:21:35.048722836 +0000 UTC m=+2475.240560448"
Sep 09 21:21:36.879902 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:36.879867    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:21:44.406692 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:44.406662    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:44.407188 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:44.406721    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:44.419023 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:44.418996    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:21:45.084546 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:21:45.084516    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:22:02.768982 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:02.768944    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"]
Sep 09 21:22:02.787108 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:02.787079    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:03.511961 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:03.511932    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"]
Sep 09 21:22:03.538185 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:03.538156    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:22:05.866490 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:05.866458    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:22:15.158931 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:15.158849    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:22:17.788634 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:17.788594    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf" podUID="31ba19cd-25b7-4e3b-9f40-67cbd687ae88" containerName="main" containerID="cri-o://d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861" gracePeriod=45
Sep 09 21:22:18.029789 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.029766    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:22:18.121129 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-dshm\") pod \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") "
Sep 09 21:22:18.121129 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121093    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-model-cache\") pod \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") "
Sep 09 21:22:18.121129 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121121    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlzpv\" (UniqueName: \"kubernetes.io/projected/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-kube-api-access-qlzpv\") pod \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") "
Sep 09 21:22:18.121426 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121162    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-home\") pod \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") "
Sep 09 21:22:18.121426 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121191    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tmp-dir\") pod \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") "
Sep 09 21:22:18.121426 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121251    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tls-certs\") pod \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\" (UID: \"31ba19cd-25b7-4e3b-9f40-67cbd687ae88\") "
Sep 09 21:22:18.121426 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121413    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-model-cache" (OuterVolumeSpecName: "model-cache") pod "31ba19cd-25b7-4e3b-9f40-67cbd687ae88" (UID: "31ba19cd-25b7-4e3b-9f40-67cbd687ae88"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:18.121620 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121481    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-home" (OuterVolumeSpecName: "home") pod "31ba19cd-25b7-4e3b-9f40-67cbd687ae88" (UID: "31ba19cd-25b7-4e3b-9f40-67cbd687ae88"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:18.121620 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121500    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "31ba19cd-25b7-4e3b-9f40-67cbd687ae88" (UID: "31ba19cd-25b7-4e3b-9f40-67cbd687ae88"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:18.121620 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121577    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:18.121620 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121591    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:18.121620 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.121600    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:18.123461 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.123424    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "31ba19cd-25b7-4e3b-9f40-67cbd687ae88" (UID: "31ba19cd-25b7-4e3b-9f40-67cbd687ae88"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:18.123461 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.123433    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-dshm" (OuterVolumeSpecName: "dshm") pod "31ba19cd-25b7-4e3b-9f40-67cbd687ae88" (UID: "31ba19cd-25b7-4e3b-9f40-67cbd687ae88"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:18.123461 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.123439    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-kube-api-access-qlzpv" (OuterVolumeSpecName: "kube-api-access-qlzpv") pod "31ba19cd-25b7-4e3b-9f40-67cbd687ae88" (UID: "31ba19cd-25b7-4e3b-9f40-67cbd687ae88"). InnerVolumeSpecName "kube-api-access-qlzpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:18.221214 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.221182    2566 generic.go:356] "Generic (PLEG): container finished" podID="31ba19cd-25b7-4e3b-9f40-67cbd687ae88" containerID="d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861" exitCode=0
Sep 09 21:22:18.221353 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.221223    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"
Sep 09 21:22:18.221353 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.221259    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf" event={"ID":"31ba19cd-25b7-4e3b-9f40-67cbd687ae88","Type":"ContainerDied","Data":"d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861"}
Sep 09 21:22:18.221353 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.221301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf" event={"ID":"31ba19cd-25b7-4e3b-9f40-67cbd687ae88","Type":"ContainerDied","Data":"3d660fd4575075f72bbaf5e54980c01edae9afb28a8450e5cd7288cfe7f2dc74"}
Sep 09 21:22:18.221353 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.221335    2566 scope.go:117] "RemoveContainer" containerID="d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861"
Sep 09 21:22:18.222049 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.222028    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlzpv\" (UniqueName: \"kubernetes.io/projected/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-kube-api-access-qlzpv\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:18.222136 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.222051    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:18.222136 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.222066    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/31ba19cd-25b7-4e3b-9f40-67cbd687ae88-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:18.230356 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.230296    2566 scope.go:117] "RemoveContainer" containerID="d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861"
Sep 09 21:22:18.230603 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:22:18.230585    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861\": container with ID starting with d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861 not found: ID does not exist" containerID="d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861"
Sep 09 21:22:18.230658 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.230611    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861"} err="failed to get container status \"d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861\": rpc error: code = NotFound desc = could not find container \"d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861\": container with ID starting with d5d29f10bad38462e53b5c2f0f21393d58178f0b559a218c9a71de0ddaa01861 not found: ID does not exist"
Sep 09 21:22:18.247583 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.247562    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"]
Sep 09 21:22:18.252404 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.252384    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-4b9eccda-kserve-d79b97695-qx8lf"]
Sep 09 21:22:18.288467 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.288438    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31ba19cd-25b7-4e3b-9f40-67cbd687ae88" path="/var/lib/kubelet/pods/31ba19cd-25b7-4e3b-9f40-67cbd687ae88/volumes"
Sep 09 21:22:18.531359 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.531301    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf" podUID="1f21536b-a736-4fd6-804f-dd59462c79e2" containerName="main" containerID="cri-o://04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6" gracePeriod=45
Sep 09 21:22:18.772271 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.772251    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:22:18.927842 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.927812    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-dshm\") pod \"1f21536b-a736-4fd6-804f-dd59462c79e2\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") "
Sep 09 21:22:18.928270 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.927881    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f21536b-a736-4fd6-804f-dd59462c79e2-tls-certs\") pod \"1f21536b-a736-4fd6-804f-dd59462c79e2\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") "
Sep 09 21:22:18.928270 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.927918    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-tmp-dir\") pod \"1f21536b-a736-4fd6-804f-dd59462c79e2\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") "
Sep 09 21:22:18.928270 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.927946    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-model-cache\") pod \"1f21536b-a736-4fd6-804f-dd59462c79e2\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") "
Sep 09 21:22:18.928270 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.928003    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/1f21536b-a736-4fd6-804f-dd59462c79e2-kube-api-access-qs6qt\") pod \"1f21536b-a736-4fd6-804f-dd59462c79e2\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") "
Sep 09 21:22:18.928270 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.928027    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-home\") pod \"1f21536b-a736-4fd6-804f-dd59462c79e2\" (UID: \"1f21536b-a736-4fd6-804f-dd59462c79e2\") "
Sep 09 21:22:18.928270 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.928170    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1f21536b-a736-4fd6-804f-dd59462c79e2" (UID: "1f21536b-a736-4fd6-804f-dd59462c79e2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:18.928573 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.928379    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-tmp-dir\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:18.928573 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.928386    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-model-cache" (OuterVolumeSpecName: "model-cache") pod "1f21536b-a736-4fd6-804f-dd59462c79e2" (UID: "1f21536b-a736-4fd6-804f-dd59462c79e2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:18.928573 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.928428    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-home" (OuterVolumeSpecName: "home") pod "1f21536b-a736-4fd6-804f-dd59462c79e2" (UID: "1f21536b-a736-4fd6-804f-dd59462c79e2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:18.930116 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.930094    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f21536b-a736-4fd6-804f-dd59462c79e2-kube-api-access-qs6qt" (OuterVolumeSpecName: "kube-api-access-qs6qt") pod "1f21536b-a736-4fd6-804f-dd59462c79e2" (UID: "1f21536b-a736-4fd6-804f-dd59462c79e2"). InnerVolumeSpecName "kube-api-access-qs6qt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:18.930116 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.930101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-dshm" (OuterVolumeSpecName: "dshm") pod "1f21536b-a736-4fd6-804f-dd59462c79e2" (UID: "1f21536b-a736-4fd6-804f-dd59462c79e2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 09 21:22:18.930236 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:18.930147    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f21536b-a736-4fd6-804f-dd59462c79e2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1f21536b-a736-4fd6-804f-dd59462c79e2" (UID: "1f21536b-a736-4fd6-804f-dd59462c79e2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:19.029279 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.029240    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1f21536b-a736-4fd6-804f-dd59462c79e2-tls-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:19.029279 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.029275    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-model-cache\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:19.029279 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.029288    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs6qt\" (UniqueName: \"kubernetes.io/projected/1f21536b-a736-4fd6-804f-dd59462c79e2-kube-api-access-qs6qt\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:19.029490 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.029300    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-home\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:19.029490 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.029335    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1f21536b-a736-4fd6-804f-dd59462c79e2-dshm\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:19.227662 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.227576    2566 generic.go:356] "Generic (PLEG): container finished" podID="1f21536b-a736-4fd6-804f-dd59462c79e2" containerID="04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6" exitCode=0
Sep 09 21:22:19.227662 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.227626    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"
Sep 09 21:22:19.227662 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.227639    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf" event={"ID":"1f21536b-a736-4fd6-804f-dd59462c79e2","Type":"ContainerDied","Data":"04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6"}
Sep 09 21:22:19.227920 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.227669    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf" event={"ID":"1f21536b-a736-4fd6-804f-dd59462c79e2","Type":"ContainerDied","Data":"4402031fbc772bbbd6a934e166f1f2f7fe085f4951ed9f4ca0a0e1cae7ada6ef"}
Sep 09 21:22:19.227920 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.227685    2566 scope.go:117] "RemoveContainer" containerID="04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6"
Sep 09 21:22:19.236726 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.236707    2566 scope.go:117] "RemoveContainer" containerID="04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6"
Sep 09 21:22:19.236981 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:22:19.236959    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6\": container with ID starting with 04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6 not found: ID does not exist" containerID="04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6"
Sep 09 21:22:19.237044 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.236990    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6"} err="failed to get container status \"04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6\": rpc error: code = NotFound desc = could not find container \"04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6\": container with ID starting with 04037524df2ebfdc4bf9ed4d40246b0f1db706cda30ecd8bbe689880a0ccd1f6 not found: ID does not exist"
Sep 09 21:22:19.250664 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.250638    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"]
Sep 09 21:22:19.259166 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:19.255595    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-4b9eccda-kserve-685b769647-9z9qf"]
Sep 09 21:22:20.285545 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.285514    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f21536b-a736-4fd6-804f-dd59462c79e2" path="/var/lib/kubelet/pods/1f21536b-a736-4fd6-804f-dd59462c79e2/volumes"
Sep 09 21:22:20.364063 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.364003    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:22:20.446273 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.446232    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"]
Sep 09 21:22:20.446605 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.446592    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31ba19cd-25b7-4e3b-9f40-67cbd687ae88" containerName="main"
Sep 09 21:22:20.446652 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.446607    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="31ba19cd-25b7-4e3b-9f40-67cbd687ae88" containerName="main"
Sep 09 21:22:20.446652 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.446615    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f21536b-a736-4fd6-804f-dd59462c79e2" containerName="main"
Sep 09 21:22:20.446652 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.446621    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f21536b-a736-4fd6-804f-dd59462c79e2" containerName="main"
Sep 09 21:22:20.446754 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.446697    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f21536b-a736-4fd6-804f-dd59462c79e2" containerName="main"
Sep 09 21:22:20.446754 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.446709    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="31ba19cd-25b7-4e3b-9f40-67cbd687ae88" containerName="main"
Sep 09 21:22:20.450885 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.450862    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.463759 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.463735    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"]
Sep 09 21:22:20.542343 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.542244    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-openshift-service-ca\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.542473 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.542352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-maas-api-tls\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.542473 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.542423    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk4vx\" (UniqueName: \"kubernetes.io/projected/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-kube-api-access-wk4vx\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.542473 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.542455    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-metrics-certs\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.643095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.643054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-metrics-certs\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.643238 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.643125    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-openshift-service-ca\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.643238 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.643159    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-maas-api-tls\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.643238 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.643183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk4vx\" (UniqueName: \"kubernetes.io/projected/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-kube-api-access-wk4vx\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.643906 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.643865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-openshift-service-ca\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.645707 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.645683    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-maas-api-tls\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.645807 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.645762    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-metrics-certs\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.651521 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.651497    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk4vx\" (UniqueName: \"kubernetes.io/projected/8de004fb-d1bd-4a1f-8437-7ce34b03adc3-kube-api-access-wk4vx\") pod \"maas-api-6f8859b88f-nszp6\" (UID: \"8de004fb-d1bd-4a1f-8437-7ce34b03adc3\") " pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:20.761602 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:20.761578    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:21.043563 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:21.043508    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"]
Sep 09 21:22:21.046086 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:22:21.046053    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8de004fb_d1bd_4a1f_8437_7ce34b03adc3.slice/crio-a3968d5f99ed0483516c444debba3008fbd59fb0e5a44738aa01c4df0ff19c97 WatchSource:0}: Error finding container a3968d5f99ed0483516c444debba3008fbd59fb0e5a44738aa01c4df0ff19c97: Status 404 returned error can't find the container with id a3968d5f99ed0483516c444debba3008fbd59fb0e5a44738aa01c4df0ff19c97
Sep 09 21:22:21.241617 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:21.241552    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6" event={"ID":"8de004fb-d1bd-4a1f-8437-7ce34b03adc3","Type":"ContainerStarted","Data":"a3968d5f99ed0483516c444debba3008fbd59fb0e5a44738aa01c4df0ff19c97"}
Sep 09 21:22:22.248669 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:22.248626    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6" event={"ID":"8de004fb-d1bd-4a1f-8437-7ce34b03adc3","Type":"ContainerStarted","Data":"d045a1e7fd8d82a8591f36f51012165cab69d48d33e5fd416dc42996286266fb"}
Sep 09 21:22:22.249088 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:22.248707    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:22.274095 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:22.274050    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6" podStartSLOduration=1.599038251 podStartE2EDuration="2.274035094s" totalImagesPullingTime="674.996843ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:22:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:22:22.272187691 +0000 UTC m=+2522.464025289" watchObservedRunningTime="2026-09-09 21:22:22.274035094 +0000 UTC m=+2522.465872692"
Sep 09 21:22:27.865439 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:27.865403    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:22:28.261383 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:28.261355    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6f8859b88f-nszp6"
Sep 09 21:22:28.361026 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:28.360985    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"]
Sep 09 21:22:28.361271 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:28.361246    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k" podUID="5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" containerName="maas-api" containerID="cri-o://a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134" gracePeriod=30
Sep 09 21:22:29.605640 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.605620    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 21:22:29.714610 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.714534    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-maas-api-tls\") pod \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") "
Sep 09 21:22:29.714610 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.714599    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-openshift-service-ca\") pod \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") "
Sep 09 21:22:29.714799 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.714624    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-metrics-certs\") pod \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") "
Sep 09 21:22:29.714799 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.714654    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxh2w\" (UniqueName: \"kubernetes.io/projected/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-kube-api-access-zxh2w\") pod \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\" (UID: \"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56\") "
Sep 09 21:22:29.722664 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.715539    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" (UID: "5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 09 21:22:29.722664 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.717413    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" (UID: "5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:29.723274 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.722999    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" (UID: "5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 09 21:22:29.723527 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.723435    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-kube-api-access-zxh2w" (OuterVolumeSpecName: "kube-api-access-zxh2w") pod "5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" (UID: "5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56"). InnerVolumeSpecName "kube-api-access-zxh2w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 09 21:22:29.815918 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.815892    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxh2w\" (UniqueName: \"kubernetes.io/projected/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-kube-api-access-zxh2w\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:29.815918 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.815915    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-maas-api-tls\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:29.816064 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.815924    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-openshift-service-ca\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:29.816064 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:29.815933    2566 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56-metrics-certs\") on node \"ip-10-0-137-187.ec2.internal\" DevicePath \"\""
Sep 09 21:22:30.290431 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:30.290396    2566 generic.go:356] "Generic (PLEG): container finished" podID="5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" containerID="a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134" exitCode=0
Sep 09 21:22:30.290589 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:30.290452    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"
Sep 09 21:22:30.290589 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:30.290483    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k" event={"ID":"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56","Type":"ContainerDied","Data":"a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134"}
Sep 09 21:22:30.290589 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:30.290531    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k" event={"ID":"5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56","Type":"ContainerDied","Data":"e4eed5c52dda5a71c1aef4f7954360bdaf13118655d64943aeebb7d1965598da"}
Sep 09 21:22:30.290589 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:30.290560    2566 scope.go:117] "RemoveContainer" containerID="a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134"
Sep 09 21:22:30.300159 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:30.300140    2566 scope.go:117] "RemoveContainer" containerID="a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134"
Sep 09 21:22:30.300453 ip-10-0-137-187 kubenswrapper[2566]: E0909 21:22:30.300432    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134\": container with ID starting with a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134 not found: ID does not exist" containerID="a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134"
Sep 09 21:22:30.300565 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:30.300459    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134"} err="failed to get container status \"a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134\": rpc error: code = NotFound desc = could not find container \"a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134\": container with ID starting with a0cbfffab3a54bb5364834028c6c6d3c4a26aaf3cb147e5584b37c6c46eac134 not found: ID does not exist"
Sep 09 21:22:30.310924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:30.310902    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"]
Sep 09 21:22:30.315709 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:30.315687    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6ffbd7f869-tl22k"]
Sep 09 21:22:32.285697 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:32.285667    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" path="/var/lib/kubelet/pods/5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56/volumes"
Sep 09 21:22:34.788121 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:34.788084    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:22:40.561770 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:40.561733    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:22:47.104425 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:47.104387    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-gkth4"]
Sep 09 21:22:55.574303 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:55.574272    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-k79s4_3eb13ab3-9716-48f0-adfe-be6b20e57029/manager/0.log"
Sep 09 21:22:55.699407 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:55.699373    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vfgww_8d70471c-1528-4fbc-ad80-10f52b4dda5e/manager/0.log"
Sep 09 21:22:55.839395 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:55.839300    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86f6d44ffc-c2v9z_3329450d-370d-40e1-aef5-f9e2fbbd2224/manager/0.log"
Sep 09 21:22:56.377819 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:56.377792    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d46659dfc-tz2zw_d454cced-f2f7-47c8-bb0a-6fb136622c54/copy-manifests/0.log"
Sep 09 21:22:56.385609 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:56.385582    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d46659dfc-tz2zw_d454cced-f2f7-47c8-bb0a-6fb136622c54/manager/0.log"
Sep 09 21:22:56.964292 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:56.964259    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6f8859b88f-nszp6_8de004fb-d1bd-4a1f-8437-7ce34b03adc3/maas-api/0.log"
Sep 09 21:22:57.092649 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:57.092613    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29816460-tfxzz_099ebc12-123b-43d9-8208-f661340dcadd/cleanup/0.log"
Sep 09 21:22:58.187938 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:58.187908    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-f94rx_e4813fee-7416-4e1c-a7ff-8f990a2613b1/manager/0.log"
Sep 09 21:22:58.298021 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:58.297981    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-skwn8_4b719d34-29d8-4174-ad70-2232de060462/manager/0.log"
Sep 09 21:22:58.411240 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:58.411215    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-q58hc_2476a5e6-8c10-4cd5-a581-c52840309c25/kuadrant-console-plugin/0.log"
Sep 09 21:22:58.649568 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:58.649540    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-gkth4_d56b66e0-bc47-47f3-92bc-78ce181afb88/limitador/0.log"
Sep 09 21:22:58.770360 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:58.770327    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dfktn_a93f16ac-0405-42b4-8407-b551870fefd5/manager/0.log"
Sep 09 21:22:59.437172 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:59.437139    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c8d5d5c67-mtqtw_a175b88d-bb68-48d0-ad85-268cf209de6d/kube-auth-proxy/0.log"
Sep 09 21:22:59.552108 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:59.552078    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7mxfq_712d1861-3ac0-4548-9336-953dc2d2af00/istio-proxy/1.log"
Sep 09 21:22:59.681924 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:22:59.681895    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dhhh5_eecfe544-adc8-4087-bc4e-afd45adf8f00/payload-pre-processing/0.log"
Sep 09 21:23:00.274475 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:00.274448    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-56rnz_3b325e10-690d-4bfb-8bfa-5e3f113fa9f7/main/0.log"
Sep 09 21:23:00.609162 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:00.609071    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-tc2xg_5f767224-76fc-4652-b57e-1110eb4d8808/main/0.log"
Sep 09 21:23:00.972209 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:00.972126    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-cvp25_cb25a91f-f0ad-40db-8a12-22126a906aa1/main/0.log"
Sep 09 21:23:05.974197 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:05.974164    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dsjk4/must-gather-98ss8"]
Sep 09 21:23:05.974800 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:05.974702    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" containerName="maas-api"
Sep 09 21:23:05.974800 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:05.974721    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" containerName="maas-api"
Sep 09 21:23:05.974926 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:05.974809    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebbac2d-8a2b-4cdc-af8f-fc8b23101a56" containerName="maas-api"
Sep 09 21:23:05.977872 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:05.977852    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dsjk4/must-gather-98ss8"
Sep 09 21:23:05.981781 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:05.981759    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dsjk4\"/\"kube-root-ca.crt\""
Sep 09 21:23:05.981781 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:05.981775    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dsjk4\"/\"default-dockercfg-78ph6\""
Sep 09 21:23:05.981781 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:05.981759    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dsjk4\"/\"openshift-service-ca.crt\""
Sep 09 21:23:06.000774 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.000749    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:06.008778 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.008757    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d691d156-842f-4fc5-9321-c3cca343b0e0-must-gather-output\") pod \"must-gather-98ss8\" (UID: \"d691d156-842f-4fc5-9321-c3cca343b0e0\") " pod="openshift-must-gather-dsjk4/must-gather-98ss8"
Sep 09 21:23:06.008844 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.008812    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhn9p\" (UniqueName: \"kubernetes.io/projected/d691d156-842f-4fc5-9321-c3cca343b0e0-kube-api-access-jhn9p\") pod \"must-gather-98ss8\" (UID: \"d691d156-842f-4fc5-9321-c3cca343b0e0\") " pod="openshift-must-gather-dsjk4/must-gather-98ss8"
Sep 09 21:23:06.069219 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.069197    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dsjk4/must-gather-98ss8"]
Sep 09 21:23:06.109614 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.109595    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhn9p\" (UniqueName: \"kubernetes.io/projected/d691d156-842f-4fc5-9321-c3cca343b0e0-kube-api-access-jhn9p\") pod \"must-gather-98ss8\" (UID: \"d691d156-842f-4fc5-9321-c3cca343b0e0\") " pod="openshift-must-gather-dsjk4/must-gather-98ss8"
Sep 09 21:23:06.109709 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.109633    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d691d156-842f-4fc5-9321-c3cca343b0e0-must-gather-output\") pod \"must-gather-98ss8\" (UID: \"d691d156-842f-4fc5-9321-c3cca343b0e0\") " pod="openshift-must-gather-dsjk4/must-gather-98ss8"
Sep 09 21:23:06.109884 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.109869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d691d156-842f-4fc5-9321-c3cca343b0e0-must-gather-output\") pod \"must-gather-98ss8\" (UID: \"d691d156-842f-4fc5-9321-c3cca343b0e0\") " pod="openshift-must-gather-dsjk4/must-gather-98ss8"
Sep 09 21:23:06.118512 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.118485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhn9p\" (UniqueName: \"kubernetes.io/projected/d691d156-842f-4fc5-9321-c3cca343b0e0-kube-api-access-jhn9p\") pod \"must-gather-98ss8\" (UID: \"d691d156-842f-4fc5-9321-c3cca343b0e0\") " pod="openshift-must-gather-dsjk4/must-gather-98ss8"
Sep 09 21:23:06.134883 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.134854    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log" limit=1024
Sep 09 21:23:06.286954 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.286920    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dsjk4/must-gather-98ss8"
Sep 09 21:23:06.572671 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.572646    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dsjk4/must-gather-98ss8"]
Sep 09 21:23:06.574686 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:23:06.574650    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd691d156_842f_4fc5_9321_c3cca343b0e0.slice/crio-fe053e5d08f04d9407b425a02d9f61f61984585c31fb815679355bb28bc99f04 WatchSource:0}: Error finding container fe053e5d08f04d9407b425a02d9f61f61984585c31fb815679355bb28bc99f04: Status 404 returned error can't find the container with id fe053e5d08f04d9407b425a02d9f61f61984585c31fb815679355bb28bc99f04
Sep 09 21:23:06.576493 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:06.576478    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 09 21:23:07.456989 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:07.456957    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dsjk4/must-gather-98ss8" event={"ID":"d691d156-842f-4fc5-9321-c3cca343b0e0","Type":"ContainerStarted","Data":"b6a2927ab15b73fbebaca4f1ed742f7f4dcd1bd4f57de884bd0a76d7657d7ca0"}
Sep 09 21:23:07.457449 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:07.456999    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dsjk4/must-gather-98ss8" event={"ID":"d691d156-842f-4fc5-9321-c3cca343b0e0","Type":"ContainerStarted","Data":"fe053e5d08f04d9407b425a02d9f61f61984585c31fb815679355bb28bc99f04"}
Sep 09 21:23:08.467859 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:08.467818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dsjk4/must-gather-98ss8" event={"ID":"d691d156-842f-4fc5-9321-c3cca343b0e0","Type":"ContainerStarted","Data":"9c100499023a7c8e92f77f1b8ab96048c966ca4ce17e0afdb8d713d55ae84789"}
Sep 09 21:23:08.489623 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:08.489562    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dsjk4/must-gather-98ss8" podStartSLOduration=2.761407687 podStartE2EDuration="3.489541137s" totalImagesPullingTime="728.13345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-09 21:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:08.488818546 +0000 UTC m=+2568.680656146" watchObservedRunningTime="2026-09-09 21:23:08.489541137 +0000 UTC m=+2568.681378734"
Sep 09 21:23:09.208250 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:09.208217    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zm7nt_2fe41335-60ef-4123-99c6-0c504cf31555/global-pull-secret-syncer/0.log"
Sep 09 21:23:09.279961 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:09.279928    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-55k2q_35cb7c42-1626-4c65-bde3-0f84790c896b/konnectivity-agent/0.log"
Sep 09 21:23:09.467125 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:09.467038    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-187.ec2.internal_9a5dfa25977fd74cf93272977ce002f0/haproxy/0.log"
Sep 09 21:23:13.570327 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:13.570270    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-f94rx_e4813fee-7416-4e1c-a7ff-8f990a2613b1/manager/0.log"
Sep 09 21:23:13.607628 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:13.607592    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-skwn8_4b719d34-29d8-4174-ad70-2232de060462/manager/0.log"
Sep 09 21:23:13.647795 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:13.647759    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-q58hc_2476a5e6-8c10-4cd5-a581-c52840309c25/kuadrant-console-plugin/0.log"
Sep 09 21:23:13.728258 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:13.728225    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-gkth4_d56b66e0-bc47-47f3-92bc-78ce181afb88/limitador/0.log"
Sep 09 21:23:13.808716 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:13.808678    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-dfktn_a93f16ac-0405-42b4-8407-b551870fefd5/manager/0.log"
Sep 09 21:23:16.048921 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:16.048891    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ngv59_2d273d6b-ed54-4130-9680-f0b93a3dd791/node-exporter/0.log"
Sep 09 21:23:16.074357 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:16.074303    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ngv59_2d273d6b-ed54-4130-9680-f0b93a3dd791/kube-rbac-proxy/0.log"
Sep 09 21:23:16.100068 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:16.100036    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ngv59_2d273d6b-ed54-4130-9680-f0b93a3dd791/init-textfile/0.log"
Sep 09 21:23:18.938247 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:18.938084    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"]
Sep 09 21:23:18.942652 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:18.942192    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:18.957421 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:18.957383    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"]
Sep 09 21:23:19.043827 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.043781    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-podres\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.044026 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.043838    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4rr9\" (UniqueName: \"kubernetes.io/projected/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-kube-api-access-l4rr9\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.044026 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.043901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-lib-modules\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.044026 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.043923    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-proc\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.044026 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.043960    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-sys\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.145112 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.145059    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-podres\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.145329 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.145131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4rr9\" (UniqueName: \"kubernetes.io/projected/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-kube-api-access-l4rr9\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.145329 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.145189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-lib-modules\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.145329 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.145213    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-proc\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.145329 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.145246    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-sys\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.145581 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.145411    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-sys\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.145581 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.145543    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-podres\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.145990 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.145959    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-lib-modules\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.146105 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.146037    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-proc\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.155503 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.155470    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4rr9\" (UniqueName: \"kubernetes.io/projected/a83f91ec-947d-4e2b-8b6f-91bcbe4996c2-kube-api-access-l4rr9\") pod \"perf-node-gather-daemonset-4mmdd\" (UID: \"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2\") " pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.255497 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.255457    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:19.566125 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:19.565984    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"]
Sep 09 21:23:19.570189 ip-10-0-137-187 kubenswrapper[2566]: W0909 21:23:19.570151    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poda83f91ec_947d_4e2b_8b6f_91bcbe4996c2.slice/crio-440f21876d38f80d069de5550021a0b5a45b3b1b3c007b6d749cd65dbd94950d WatchSource:0}: Error finding container 440f21876d38f80d069de5550021a0b5a45b3b1b3c007b6d749cd65dbd94950d: Status 404 returned error can't find the container with id 440f21876d38f80d069de5550021a0b5a45b3b1b3c007b6d749cd65dbd94950d
Sep 09 21:23:20.547676 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:20.547630    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd" event={"ID":"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2","Type":"ContainerStarted","Data":"3b918fef31d9310002a73896ed59f902913fa30a49e0a42ddc11a87753a7d351"}
Sep 09 21:23:20.548167 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:20.547711    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:20.548167 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:20.547738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd" event={"ID":"a83f91ec-947d-4e2b-8b6f-91bcbe4996c2","Type":"ContainerStarted","Data":"440f21876d38f80d069de5550021a0b5a45b3b1b3c007b6d749cd65dbd94950d"}
Sep 09 21:23:20.571353 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:20.568744    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd" podStartSLOduration=2.568722648 podStartE2EDuration="2.568722648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-09 21:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-09 21:23:20.566363839 +0000 UTC m=+2580.758201456" watchObservedRunningTime="2026-09-09 21:23:20.568722648 +0000 UTC m=+2580.760560249"
Sep 09 21:23:21.991944 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:21.991915    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jhwgx_5b863620-9ac7-4433-bba7-ed209457f8cf/dns/0.log"
Sep 09 21:23:22.033287 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:22.033256    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-jhwgx_5b863620-9ac7-4433-bba7-ed209457f8cf/kube-rbac-proxy/0.log"
Sep 09 21:23:22.196003 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:22.195974    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qdsbk_f60f0088-211f-490c-8012-edec87817ca1/dns-node-resolver/0.log"
Sep 09 21:23:22.892952 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:22.892911    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-g2d92_28f00347-ee67-4e91-ac2f-0dbe68892960/node-ca/0.log"
Sep 09 21:23:24.268077 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:24.268043    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c8d5d5c67-mtqtw_a175b88d-bb68-48d0-ad85-268cf209de6d/kube-auth-proxy/0.log"
Sep 09 21:23:24.305514 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:24.305482    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7mxfq_712d1861-3ac0-4548-9336-953dc2d2af00/istio-proxy/0.log"
Sep 09 21:23:24.311236 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:24.311208    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7mxfq_712d1861-3ac0-4548-9336-953dc2d2af00/istio-proxy/1.log"
Sep 09 21:23:24.359256 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:24.359231    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dhhh5_eecfe544-adc8-4087-bc4e-afd45adf8f00/payload-pre-processing/0.log"
Sep 09 21:23:25.427367 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:25.427335    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rxdcl_e562a150-9463-4b3d-a8be-a04e8703a450/serve-healthcheck-canary/0.log"
Sep 09 21:23:26.034483 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:26.034444    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kg5pd_ac16bb0c-a835-4de8-aa64-0b9d85c2508c/kube-rbac-proxy/0.log"
Sep 09 21:23:26.059597 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:26.059571    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kg5pd_ac16bb0c-a835-4de8-aa64-0b9d85c2508c/exporter/0.log"
Sep 09 21:23:26.095757 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:26.095731    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-kg5pd_ac16bb0c-a835-4de8-aa64-0b9d85c2508c/extractor/0.log"
Sep 09 21:23:26.566396 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:26.566364    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dsjk4/perf-node-gather-daemonset-4mmdd"
Sep 09 21:23:28.751892 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:28.751861    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-k79s4_3eb13ab3-9716-48f0-adfe-be6b20e57029/manager/0.log"
Sep 09 21:23:29.345972 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:29.345937    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-vfgww_8d70471c-1528-4fbc-ad80-10f52b4dda5e/manager/0.log"
Sep 09 21:23:29.425982 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:29.425950    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-86f6d44ffc-c2v9z_3329450d-370d-40e1-aef5-f9e2fbbd2224/manager/0.log"
Sep 09 21:23:29.803586 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:29.803553    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d46659dfc-tz2zw_d454cced-f2f7-47c8-bb0a-6fb136622c54/manager/0.log"
Sep 09 21:23:29.845177 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:29.845156    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d46659dfc-tz2zw_d454cced-f2f7-47c8-bb0a-6fb136622c54/copy-manifests/0.log"
Sep 09 21:23:30.016531 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:30.016496    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c8d5d5c67-mtqtw_a175b88d-bb68-48d0-ad85-268cf209de6d/kube-auth-proxy/0.log"
Sep 09 21:23:30.046416 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:30.046387    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7mxfq_712d1861-3ac0-4548-9336-953dc2d2af00/istio-proxy/0.log"
Sep 09 21:23:30.049974 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:30.049944    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7mxfq_712d1861-3ac0-4548-9336-953dc2d2af00/istio-proxy/1.log"
Sep 09 21:23:30.093989 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:30.093906    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dhhh5_eecfe544-adc8-4087-bc4e-afd45adf8f00/payload-pre-processing/0.log"
Sep 09 21:23:32.090190 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:32.090160    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6c8d5d5c67-mtqtw_a175b88d-bb68-48d0-ad85-268cf209de6d/kube-auth-proxy/0.log"
Sep 09 21:23:32.120933 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:32.120905    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7mxfq_712d1861-3ac0-4548-9336-953dc2d2af00/istio-proxy/0.log"
Sep 09 21:23:32.126785 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:32.126757    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-7mxfq_712d1861-3ac0-4548-9336-953dc2d2af00/istio-proxy/1.log"
Sep 09 21:23:32.162942 ip-10-0-137-187 kubenswrapper[2566]: I0909 21:23:32.162912    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-dhhh5_eecfe544-adc8-4087-bc4e-afd45adf8f00/payload-pre-processing/0.log"